From 9909fc1882a7e29a5bed7c8faee09a0a96f1c5d6 Mon Sep 17 00:00:00 2001 From: Gerry Nelson <45608078+gerrynelson63@users.noreply.github.com> Date: Sun, 11 Apr 2021 10:22:22 -0400 Subject: [PATCH] Viya4 v1 (#75) * add getosixidentity.py * Add new tool that is in master to viya4_v1 (#73) * Viya4 v1 (#64) * add properties file * add get properties function * add get of properties to listreports * update listreports * fix * add insepct * test with foldertree * update properties * test export folder tree * exportfoldertree * make cli location and command dynamic * add imports * fix * add os * add all option * updates * update import packages * update import packages * a few more sas-viya changes * add function * showsetup * make targetname optional * update * update * Update sharedfunctions.py added get_valid_filename * Update snapshotreports.py * Update updatepreferences.py upped the limit on all users * Update INSTALL.md * Update INSTALL.md * Update application.properties Update the default cli name for Viya 3.x * (PSETRIAGE-1406) import notification templates (#70) This is to compensate for lack of notification template editor and template VLI to laod templates using template service. * PSETRIAGE-1406) import notification templates (#71) Some bug fixes * add getosixidentity.py (#72) Co-authored-by: Lori Small * Update application.properties * add importcaslibs.py * addd +x * fix cass comment * fix tabs * update message * update message * message when no files * fix indent * Update importcaslibs.py * Update importcaslibs.py * +x * update * add comments to setup.py * updates to doc * +x * update readme * temporary set to sas-admin Co-authored-by: Lori Small --- INSTALL.md | 67 ++++++++++++++++++++----------- README.md | 51 +++++++++++++---------- importcaslibs.py | 91 ++++++++++++++++++++++++++++++++++++++++++ importconfiguration.py | 80 +++++++++++++++++++++++++++++++++++++ setup.py | 73 +++++++++++++++++++++++++++++++++ 5 files changed, 318 insertions(+), 44 deletions(-) create mode 100755 importcaslibs.py create mode 100755 importconfiguration.py create mode 100755 setup.py diff --git a/INSTALL.md b/INSTALL.md index 38cf4a2..12e747c 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,58 +1,79 @@ -**pyviyatools Install Intructions** +# pyviyatools Install Intructions -The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. +* [Install](#install) + * [Viya 3.x](#viya-3x) + * [Viya 4](#viya-4) + * [Configure](#configure) +* [Authentication](#authentication) -**INSTALL** +The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. A subset of the tools also call the SAS Administration CLI + +## Install The tools are a package of files and should be downloaded as such. The individual files are not useable without the package. -The tools should be installed on the same machine that hosts the Viya command-line interfaces(CLI). The following command will install a copy of the tools in a sub-directory(pyviyatools) of the current directory. +The tools should be installed on the same machine that hosts the Viya command-line interfaces(CLI). The following commands will install a copy of the tools in a sub-directory(pyviyatools) of the current directory. + +### Viya 3.x + +*git clone https://github.com/sassoftware/pyviyatools.git* + +### Viya 4 *git clone https://github.com/sassoftware/pyviyatools.git* -NOTE: to use the tools with Viya 4 clone the viya4_v1 branch +cd $install-dir + +./setup.py -*git clone https://github.com/sassoftware/pyviyatools.git -b viya4_v1* +### Configure -**Configure** +What does .setup.py do? -The application.properties file contains the default location of the sas-admin or viya cli. Edit this file to match the location of the cli and the cli name in your environment. +Viya 3.5 the SAS Administration cli executable is sas-admin and the Viya 4 cli is sas-viya. -The default values for Viya 3.x are: +The application.properties file contains the name and location of the SAS Administation cli. -sascli.location=/opt/sas/viya/home/bin/ -sascli.executable=sas-admin -The default values for Viya 4.x are: +The default application properties in the repository stores the values for Viya 3.x -sascli.location=/opt/sas/viya/home/bin/ -sascli.executable=sas-viya +* sascli.location=/opt/sas/viya/home/bin/ +* sascli.executable=sas-admin -**Authenticate** +To configure the tools for Viya 4, or if you have your cli installed in a non-default location use ./setup.py. For example: + +**./setup.py --clilocation /opt/bin --cliexecutable sas-viya** + +If you pass no parameters the defaults for the function are the Viya 4 default values. + +* sascli.location=/opt/sas/viya/home/bin/ +* sascli.executable=sas-viya + +## Authentication The pyviya tools use the sas-admin auth CLI to authenticate to Viya. To use the tool you must create a profile and authenticate. This process is documented in the SAS Viya Administration guide here. http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.3&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en -If your environment is enabled for Transport Layer Security (TLS), you must set the SSL_CERT_FILE environment variable to the path location of the trustedcerts.pem file (if using the SAS default truststore) -or the path location of your site-signed certificate (if using an internal truststore). +If your environment is enabled for Transport Layer Security (TLS), you must set the SSL_CERT_FILE environment variable to the path location of the trustedcerts.pem file (if using the SAS default truststore) +or the path location of your site-signed certificate (if using an internal truststore). In addition you may need to set REQUESTS_CA_BUNDLE to the certificate location so that the requests library can find the certificates. The tool will automatically use the default profile. - -1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's -2. Run */opt/sas/viya/home/bin/sas-admin profile init* and when prompter enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. + +1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's +2. Run */opt/sas/viya/home/bin/sas-admin profile init* and when prompter enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. 3. After you create the profile there are two options to authenticate * Run */opt/sas/viya/home/bin/sas-admin auth login* to authenticate to Viya enter the userid and password of the SAS Administrator when prompted. - * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) + * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) The CLI allows for multiple profiles. To use a profile other than the default profile, for example newprofile 1. Create a named profile *sas-admin --profile newprofile profile init* 2. Logon with the named profile *sas-admin --profile newprofile auth login* -3. Set the SAS_CLI_PROFILE environment variable to the name of the profile +3. Set the SAS_CLI_PROFILE environment variable to the name of the profile * LINUX: *export SAS_CLI_PROFILE=newprofile* * WINDOWS: *set SAS_CLI_PROFILE=newprofile* @@ -68,7 +89,7 @@ The tools are self-documenting, for help on any tool call the tool passing -h or 1. Execute a test call. Change directory to your install directory and run: - *callrestapi.py -e /folders/folders -m get* + *callrestapi.py -e /folders/folders -m get* The call should return the JSON for the folders in the viya deployment. Be patient it might take a few seconds the first time. diff --git a/README.md b/README.md index e41e91a..62b4b01 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,22 @@ # Python Tools for SAS Viya -The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. The tools are designed to be used in conjunction with the sas-admin command line interfaces(CLI). - +The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. The tools are designed to be used in conjunction with the SAS Administration command line interfaces(CLI). A subset of the tools also make calls to the cli. + +* [Getting Started](#getting-started) + * [Documentation](#documentation) + * [Prerequisites](#prerequisites) + * [Installing](#installing) + * [Running](#running) + * [Creating a Profile and Logging on](#creating-a-profile-and-logging-on) +* [Troubelshooting](#troubelshooting) +* [Developing with the existing functions](#developing-with-the-existing-functions) +* [Contributing](#contributing) +* [License](#license) ## Getting Started ### Documentation -* SAS Open API's documented : https://developer.sas.com/apis/rest/ +* SAS Open API's documented : https://developer.sas.com/apis/rest/ * Sample REST API calls : https://github.com/sassoftware/devsascom-rest-api-samples Some other useful links @@ -17,7 +27,7 @@ Some other useful links ### Prerequisites -The tools require either python 2 or python 3. +The tools will work with either python 2 or python 3. The following python libraries are used: @@ -34,33 +44,33 @@ The following python libraries are used: ### Installing -Please use the installation intructions in the file [INSTALL.md](INSTALL.md) +Please use the installation intructions in the file [INSTALL.md](INSTALL.md) ### Running -The pyviya tools use the sas-admin auth CLI to authenticate to Viya. To use the tool you must create a profile and authenticate. +The pyviya tools use the SAS Administratin CLI to authenticate to Viya. To use the tool you must create a profile and authenticate. -This process is documented in the SAS Viya Administration guide here: +This process is documented in the SAS Viya Administration guide here: -* Viya 3.3: http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.3&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en +* Viya 3.3: http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.3&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en * Viya 3.4: http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.4&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en * Viya 3.5: https://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.5&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en #### Creating a Profile and Logging on The tool will automatically use the default profile. - -1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's -2. Run `/opt/sas/viya/home/bin/sas-admin profile init` and when prompted enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. + +1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's +2. Run `/opt/sas/viya/home/bin/sas-admin profile init` and when prompted enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. 3. After you create the profile there are two options to authenticate * Run `/opt/sas/viya/home/bin/sas-admin auth login` to authenticate to Viya enter the userid and password of the SAS Administrator when prompted. - * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) + * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) The CLI allows for multiple profiles. To use a profile other than the default profile, for example newprofile 1. Create a named profile *sas-admin --profile newprofile profile init* 2. Logon with the named profile *sas-admin --profile newprofile auth login* -3. Set the SAS_CLI_PROFILE environment variable to the name of the profile +3. Set the SAS_CLI_PROFILE environment variable to the name of the profile * LINUX: *export SAS_CLI_PROFILE=newprofile* * WINDOWS: *set SAS_CLI_PROFILE=newprofile* @@ -71,21 +81,21 @@ The CLI allows for multiple profiles. To use a profile other than the default pr * WINDOWS: *set SAS_CLI_PROFILE=* -The tools are self-documenting, for help on any tool call the tool passing -h or --help. +The tools are self-documenting, for help on any tool call the tool passing -h or --help. python *\\* -h **Available Tools** -**callrestapi** +**callrestapi** callrestapi is a general tool, and the building block for all the other tools. callrestapi will call the Viya REST API and return json or optionally a simplified output. It can call any viya REST method. (Like a curl command). *usage: callrestapi.py [-h] -e ENDPOINT -m {get,put,post,delete} [-i INPUTFILE] [-a ACCEPTTYPE] [-c CONTENTTYPE] [-o {csv,json,simple}]* - + You must pass a method and endpoint. You can optionally pass json, content type headers or the -t flag to change output from json to basic text. **List of some of the Additional Tools Available** @@ -106,7 +116,7 @@ Additional tools provide more complex functionality by combining multiple calls * **loginviauthinfo.py** use an authinfo file to authenticate to the CLI * **updateprefences.py** update preferences for a user or group of users * **updatedomain.py** load a set of userids and passwords to a Viya domain from a csv file -* **createfolders.py** create a set of Viya folders from a csv file +* **createfolders.py** create a set of Viya folders from a csv file * **explainaccess.py** explains access for a folder, object or service endpoint * **getpath.py** return path of folder, report, or other object in folder * **listmemberswithpath.py** lists members of a folder, recursively if desired @@ -121,12 +131,11 @@ Additional tools provide more complex functionality by combining multiple calls Check back for additional tools and if you build a tool feel free to contribute it to the collection. - -**Examples** -Most of the tools have examples of usage documented in [EXAMPLES.md](EXAMPLES.md) +**Examples** -**Troubleshooting** +Most of the tools have examples of usage documented in [EXAMPLES.md](EXAMPLES.md) +## Troubelshooting The most common problem is an expired access token. You may see a message like: diff --git a/importcaslibs.py b/importcaslibs.py new file mode 100755 index 0000000..2bbf3e5 --- /dev/null +++ b/importcaslibs.py @@ -0,0 +1,91 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +# +# importcaslibs.py +# April 2021 +# +# Pass in a directory and this tool will import all the json files in the directory. It depends on the admin CLI +# The json files should be standard caslib definitions +# +# File format +#{ +#"attributes": { "active": false, "personal": false, "subDirs": false}, +#"description": "", +#"name": "Sales2", +# "path": "/tmp/sales", +# "scope": "global", +# "server": "cas-shared-default", +# "type": "PATH" +#} +# +# +# Change History +# +# Copyright © 2019, SAS Institute Inc., Cary, NC, USA. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS +# OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +# +# +# Import Python modules +import argparse, sys, subprocess, os, json +from sharedfunctions import callrestapi, getapplicationproperties + +# get cli location from properties +propertylist=getapplicationproperties() + +clidir=propertylist["sascli.location"] +cliexe=propertylist["sascli.executable"] + +clicommand=os.path.join(clidir,cliexe) + +# get input parameters +parser = argparse.ArgumentParser(description="Import JSON files that define path-based CASLIBS from directory. All json files in directory will be imported.") +parser.add_argument("-d","--directory", help="Directory that contains JSON caslib dfinition files to import",required='True') +parser.add_argument("-q","--quiet", help="Suppress the are you sure prompt.", action='store_true') +args= parser.parse_args() +basedir=args.directory +quietmode=args.quiet + +# get python version +version=int(str(sys.version_info[0])) + +# if the quiet mode flag is not passed then prompt to continue +if not quietmode: + + if version > 2: + areyousure=input("WARNING: Are you sure? (Y)") + else: + areyousure=raw_input("WARNING: Are you sure? (Y)") +else: + areyousure="Y" + +if areyousure.upper() =='Y': + + # check that directory exists + if os.path.isdir(basedir): + + # loop files in the directory + for filename in os.listdir( basedir ): + + # only process json files + if filename.lower().endswith('.json'): + command=clicommand+' cas caslibs create path --source-file '+os.path.join(basedir,filename) + print(command) + subprocess.call(command, shell=True) + + print("NOTE: Viya Caslib imported attempted from json file "+filename+" in directory "+basedir ) + + else: print("ERROR: Directory does not exist") +else: + print("NOTE: Operation cancelled") + + + + + + + diff --git a/importconfiguration.py b/importconfiguration.py new file mode 100755 index 0000000..a427fce --- /dev/null +++ b/importconfiguration.py @@ -0,0 +1,80 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +# +# importconfiguration.py +# April 2021 +# +# Pass in a directory and this tool will import all the json files in the directory. It depends on the admin CLI +# The json files should be standard viya configuration definitions +# +# +# Change History +# +# Copyright © 2019, SAS Institute Inc., Cary, NC, USA. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the License); you may not use this file except in compliance with the License. +# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS +# OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. +# +# +# Import Python modules +import argparse, sys, subprocess, os, json +from sharedfunctions import callrestapi, getapplicationproperties + +# get cli location from properties +propertylist=getapplicationproperties() + +clidir=propertylist["sascli.location"] +cliexe=propertylist["sascli.executable"] + +clicommand=os.path.join(clidir,cliexe) + +# get input parameters +parser = argparse.ArgumentParser(description="Import JSON files that update Viya configuration. All json files in directory will be imported.") +parser.add_argument("-d","--directory", help="Directory that contains JSON configuration definition files to import",required='True') +parser.add_argument("-q","--quiet", help="Suppress the are you sure prompt.", action='store_true') +args= parser.parse_args() +basedir=args.directory +quietmode=args.quiet + +# get python version +version=int(str(sys.version_info[0])) + +# if the quiet mode flag is not passed then prompt to continue +if not quietmode: + + if version > 2: + areyousure=input("WARNING: Are you sure? (Y)") + else: + areyousure=raw_input("WARNING: Are you sure? (Y)") +else: + areyousure="Y" + +if areyousure.upper() =='Y': + + # check that directory exists + if os.path.isdir(basedir): + + # loop files in the directory + for filename in os.listdir( basedir ): + + # only process json files + if filename.lower().endswith('.json'): + command=clicommand+' configuration configurations update --file '+os.path.join(basedir,filename) + print(command) + subprocess.call(command, shell=True) + + print("NOTE: Configuration import attempted from json file "+filename+" in directory "+basedir ) + + else: print("ERROR: Directory does not exist") +else: + print("NOTE: Operation cancelled") + + + + + + + diff --git a/setup.py b/setup.py new file mode 100755 index 0000000..79a238a --- /dev/null +++ b/setup.py @@ -0,0 +1,73 @@ +#!/usr/bin/python +# -*- coding: utf-8 -*- +# +# setup.py +# December 2020 +# +# Updates the application.properties file with the location and name of the administration cli +# The default application properties stores the values for Viya 3.x +# This ensures bacward compatibility for Viya 3 users. +# +# sascli.location=/opt/sas/viya/home/bin/ +# sascli.executable=sas-admin +# +# The defaults for the function are the Viya 4 values +# +# sascli.location=/opt/sas/viya/home/bin/ +# sascli.executable=sas-viya +# +# As a result you only need to run setup.sh if you clone master and are using viya4 +# +# .setup.py will set the default values for Viya 4 +# OR +# ./setup.py --clilocation /opt/bin --cliexecutable sas-viya +# +# +# Copyright © 2018, SAS Institute Inc., Cary, NC, USA. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the License); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# Import Python modules + +import argparse, sys, configparser, os, inspect + +from sharedfunctions import getapplicationproperties +from configobj import ConfigObj + +# get python version +version=int(str(sys.version_info[0])) + +# get input parameters +parser = argparse.ArgumentParser(description="Setup pyviyatools") +parser.add_argument("--clilocation", help="Enter the full path to the viya cli.",default="/opt/sas/viya/home/bin/") +parser.add_argument("--cliexecutable", help="Enter the name of the viya cli executable.", default="sas-viya") +args = parser.parse_args() + +clilocation=args.clilocation +cliexecutable=args.cliexecutable + +thepath=os.path.split(inspect.getsourcefile(lambda:0)) +install_dir=thepath[0] +prop_file=os.path.join(install_dir, "application.properties") + +print ("Note: updating "+prop_file) + +dict={"sascli.location":clilocation,"sascli.executable":cliexecutable} + +print("NOTE: configuration values set to ", dict ) + +config = ConfigObj(dict) +config.filename = prop_file + +config.write() +