diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 0000000..97809c9 --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,19 @@ +# How to Contribute + +We'd love to accept your patches and contributions to this project. There are +just a few small guidelines you need to follow. + +## Contributor License Agreement + +Contributions to this project must be accompanied by a signed +[Contributor Agreement](ContributorAgreement.txt). +You (or your employer) retain the copyright to your contribution, +this simply gives us permission to use and redistribute your contributions as +part of the project. + +## Code reviews + +All submissions, including submissions by project members, require review. We +use GitHub pull requests for this purpose. Consult +[GitHub Help](https://help.github.com/articles/about-pull-requests/) for more +information on using pull requests. diff --git a/INSTALL.md b/INSTALL.md new file mode 100644 index 0000000..06f4eda --- /dev/null +++ b/INSTALL.md @@ -0,0 +1,68 @@ +**pyviyatools Install Intructions** + +The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. + +**INSTALL** + +The tools are a package of files and should be downloaded as such. The individual files are not useable without the package. + +The tools should be installed on the same machine that hosts the Viya command-line interfaces(CLI). + +1. Under [pyviyatools] (https://gitlab.sas.com/sasadmintools/viya/pyviyatools)select the **Download** icon (looks like a cloud with a down arrow) +2. Select **Download tar.gz** +3. In Linux terminal window change directory to the location where you downloaded the tar file +4. Use the following command to install the tools. In the filename replace *????* with the alpha-numeric string in your file and the directory */mydir/pyviyatools* with an existing directory where you would like to install the tools then execute: + + *tar -xvf "pyviyatools-master-????.tar.gz" -C /mydir/pyviyatools --strip-components=1* + +5. Add the execute permission to scripts by running + + *chmod ug+x /mydir/pyviyatools/** + +**Authenticate** + +The pyviya tools use the sas-admin auth CLI to authenticate to Viya. To use the tool you must create a profile and authenticate. This process is documented in the SAS Viya Administration guide here. + +http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.3&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en + + +If your environment is enabled for Transport Layer Security (TLS), you must set the SSL_CERT_FILE environment variable to the path location of the trustedcerts.pem file (if using the SAS default truststore) +or the path location of your site-signed certificate (if using an internal truststore). + +In addition you may need to set REQUESTS_CA_BUNDLE to the certificate location so that the requests library can find the certificates. + +The tool will automatically use the default profile. + +1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's +2. Run */opt/sas/viya/home/bin/sas-admin profile init* and when prompter enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. +3. After you create the profile there are two options to authenticate + * Run */opt/sas/viya/home/bin/sas-admin auth login* to authenticate to Viya enter the userid and password of the SAS Administrator when prompted. + * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) + +The CLI allows for multiple profiles. To use a profile other than the default profile, for example newprofile + +1. Create a named profile *sas-admin --profile newprofile profile init* +2. Logon with the named profile *sas-admin --profile newprofile auth login* +3. Set the SAS_CLI_PROFILE environment variable to the name of the profile +* LINUX: *export SAS_CLI_PROFILE=newprofile* +* WINDOWS: *set SAS_CLI_PROFILE=newprofile* + +4. To revert to using the default variable reset the environment variable + +* LINUX: *unset SAS_CLI_PROFILE* +* WINDOWS: *set SAS_CLI_PROFILE=* + +The tools are self-documenting, for help on any tool call the tool passing -h or --help. + + +**TEST** + +1. Execute a test call. Change directory to your install directory and run: + + *callrestapi.py -e /folders/folders -m get* + +The call should return the JSON for the folders in the viya deployment. Be patient it might take a few seconds the first time. + +2. To see the current setup, including python and package versions, environment variable settings, profile information and current user. Run + + *showsetup.py* \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..82fc73b --- /dev/null +++ b/README.md @@ -0,0 +1,284 @@ +# Python Tools for SAS Viya + +The pyviyatools are a set of command-line tools that call the SAS Viya REST API's from python. The tools can be used to make direct calls to any rest-endpoint (like a CURL command) or to build additional tools that make multiple rest calls to provide more complex functionality. The tools are designed to be used in conjunction with the sas-admin command line interfaces(CLI). + +## Getting Started + +### Documentation + +You can find the SAS Open API's documented: https://developer.sas.com/apis/rest/ + +Some other useful links + +* https://developer.sas.com/apis/rest/#filters +* https://developer.sas.com/apis/rest/#pagination +* https://developer.sas.com/apis/rest/#sorting + +### Prerequisites + +The tools requires either python 2 or python 3. + +The following python libraries are used: + +* requests http://docs.python-requests.org/en/master/ +* sys +* json +* os +* netrc +* subprocess +* platform +* argeparse + + +### Installing + +Please use the installation intructions in the file INSTALL.md [INSTALL.md](INSTALL.md) + +### Running + +The pyviya tools use the sas-admin auth CLI to authenticate to Viya. To use the tool you must create a profile and authenticate. + +This process is documented in the SAS Viya Administration guide here. + +http://documentation.sas.com/?cdcId=calcdc&cdcVersion=3.3&docsetId=calcli&docsetTarget=n1e2dehluji7jon1gk69yggc6i28.htm&locale=en + +#### Creating a Profile and Logging on + +The tool will automatically use the default profile. + +1. To create a profile as a SAS Administrator logon to the machine that contains the Viya CLI's +2. Run */opt/sas/viya/home/bin/sas-admin profile init* and when prompter enter the base endpoint of your Viya server for example: http://myviyaserver.blah.com. You may enter your personal preference at the other prompts. +3. After you create the profile there are two options to authenticate + * Run */opt/sas/viya/home/bin/sas-admin auth login* to authenticate to Viya enter the userid and password of the SAS Administrator when prompted. + * Create an .authinfo file in your home directory with your userid and password and use **loginviauthinfo.py** to authenticate with the credentials in the file (you can use different authinfo files with the -f option) + +The CLI allows for multiple profiles. To use a profile other than the default profile, for example newprofile + +1. Create a named profile *sas-admin --profile newprofile profile init* +2. Logon with the named profile *sas-admin --profile newprofile auth login* +3. Set the SAS_CLI_PROFILE environment variable to the name of the profile +* LINUX: *export SAS_CLI_PROFILE=newprofile* +* WINDOWS: *set SAS_CLI_PROFILE=newprofile* + +4. When the SAS_CLI_PROFILE is set the tool will use the profile stored in the variable +5. To revert to using the default variable reset the environment variable + +* LINUX: *unset SAS_CLI_PROFILE* +* WINDOWS: *set SAS_CLI_PROFILE=* + + +The tools are self-documenting, for help on any tool call the tool passing -h or --help. + +python *\\* -h + + +**Available Tools** + +**callrestapi** + +callrestapi is a general tool, and the building block for all the other tools. + +callrestapi will call the Viya REST API and return json or optionally a simplified output. It can call any viya REST method. (Like a curl command). + +*usage: callrestapi.py [-h] -e ENDPOINT -m {get,put,post,delete} [-i INPUTFILE] [-a ACCEPTTYPE] [-c CONTENTTYPE] [-o {csv,json,simple}]* + +You must pass a method and endpoint. You can optionally pass json, content type headers or the -t flag to change output from json to basic text. + +**List of some of the Additional Tools Available** + +Additional tools provide more complex functionality by combining multiple calls to the callrestapi function, and post-processing the outpuit that is returned. + +* **getfolderid** returns the id of the folder based on the full folder path +* **deletefolder** deletes a folder based on the full folder path +* **deletefolderandcontent** deletes a folder and any reports that are stored in the folder +* **movecontent** moves the content from a source to a target folder +* **getconfigurationproperties** lists the name/value pairs of a configuration +* **testfolderaccess** tests if a user or group has access to a folder +* **createbinarybackup.py** creates a binary backup job +* **createdomain.py** creates an authentication domain +* **listrules.py** list authorization rules subset on a principal and/or a uri +* **loginviauthinfo.py** use an authinfo file to authenticate to the CLI +* **updateprefences.py** update preferences for a user or group of users +* **updatedomain.py** Load a set of userids and passwords to a Viya domain from a csv file +* **createfolders.py** Create a set of Viya folders from a csv file +* **explainaccess.py** Explains access for a folder, object or service endpoint + +Check back for additional tools and if you build a tool feel free to contribute it to the collection. + +## Examples + +The example calls assume you are in the directory where you installed the tools. If you are not you can prepemd the directory to the filename, for example /opt/pyviyatools/callrestapi.py +If you are using the tool on windows the syntax is: + +*python toolname.py* + +The following examples are Linux. + +\# Login to Viya using default authinfo file ~/.authinfo +*./loginviauthinfo.py* + +\# Login to Viya using an authinfo file +*./loginviauthinfo.py -f ~/.authinfo_geladm* + +\#return all the rest calls that can be made to the folders endpoint +*./callrestapi.py -e /folders -m get* + +\#return the json for all the folders/folders +*./callrestapi.py -e /folders/folders -m get* + +\#return simple text for all the folders/folders +*./callrestapi.py -e /folders/folders -m get -o simple* + +\#rest calls often limit the results returned the text output will tell you returned and total available items +\#in this call set a limit above the total items to see everything +*./callrestapi.py -e /folders/folders?limit=500 -m get -o simple* + + +\#return the json for all the identities +*./callrestapi.py -e /identities/identities -m get* + +\#return the json for all the identities output to a file +*./callrestapi.py -e /identities/identities -m get > identities.json* + +\#refresh the identities cache +*./callrestapi.py -e /identities/userCount -m get* +*./callrestapi.py -e /identities/cache/refreshes -m post* + +\# pass the folder path and return the folder id and uri +*./getfolderid.py -f /gelcontent* + +\# delete a folder based on its path +*./deletefolder.py -f /gelcontent* + +\#return a set of configuration properties +*./getconfigurationproperties.py -c sas.identities.providers.ldap.user -o simple* + +\#create a domain using createdomain +*./createdomain.py -t password -d test -u sasadm -p lnxsas -g "SASAdministrators,HR ,Sales"* + +\#Update an existing domain to add credentials from a csv file +*./updatedomain.py -d LASRAuth -f /tmp/myusers.csv* + +csv file format +no header row +column1 is userid +column2 is password +column3 is identity +column4 is identity type (user or group) + +For example: +myuserid,mypass,Sales,group +acct1,pw1,sasadm,user + +\#create a domain using callrestapi. Last part of endpoint is domain name +*./callrestapi.py -e /credentials/domains/\ -m post -i domain.json* + +INPUT JSON must be formatted as the endpoint expects. Example: + +{ + "id": "\", + "type": "password" +} + +\#test folder access +*./root/admin/pyviyatools/testFolderAccess.py -f '/gelcontent' -p Sales -m read -s grant -q* + +\#return basic content using accept response type +*./callrestapi.py -m get -e /identities/users/sasadm -a "application/vnd.sas.identity.basic+json"* + +\# Display all sasadministrator rules +*./listrules.py --p SASadministrators -o simple* + +\# Display all rules that contain SASVisual in the URI +*./listrules.py -u SASVisual -o simple* + +\# Create folders from a CSV file +*./createfolders.py" -f /tmp/newfolders.csv* + +FORMAT OF CSV file folder path (parents must exist), description +/RnD, Folder under root for R&D +/RnD/reports, reports +/RnD/analysis, analysis +/RnD/data plans, data plans +/temp,My temporary folder +/temp/mystuff, sub-folder + +\# Update the theme for the user sasadm +*./updatepreferences.py -t user -tn sasadm -pi OpenUI.Theme.Default -pv sas_hcb* + + +\#Explain direct and indirect permissions on the folder /folderA/folderB, no header row. For folders, conveyed permissions are shown by default. +*./explainaccess.py -f /folderA/folderB* + +\#As above but for a specific user named Heather +*./explainaccess.py -f /folderA/folderB -n Heather -t user* + +\#As above with a header row +*./explainaccess.py -f /folderA/folderB --header* + +\#As above with a header row and the folder path, which is useful if you concatenate sets of results in one file +*./explainaccess.py -f /folderA/folderB -p --header* + +\#As above showing only rows which include a direct grant or prohibit +*./explainaccess.py -f /folderA/folderB --direct_only* + +\#Explain direct and indirect permissions on a service endpoint. Note in the results that there are no conveyed permissions. By default they are not shown for URIs. +*./explainaccess.py -u /SASEnvironmentManager/dashboard* + +\#As above but including a header row and the create permission, which is relevant for services but not for folders and other objects +*./explainaccess.py -u /SASEnvironmentManager/dashboard --header -l read update delete secure add remove create* + +\#Explain direct and indirect permissions on a report, reducing the permissions reported to just read, update, delete and secure, since none of add, remove or create are applicable to a report. +*./explainaccess.py -u /reports/reports/*folder_id *--header -l read update delete secure* + +\#Explain direct and indirect permissions on a folder expressed as a URI. Keep the default permissions list, but for completeness we must also specify -c true to request conveyed permissions be displayed, as they are not displayed by default for URIs. +*./explainaccess.py -u /folders/folders/*folder_id *--header -p -c true* + +**Troubleshooting** + +The most common problem is an expired access token. You may see a message like: + + {"errorCode":0,"message":"Cannot convert access token to JSON","details":["traceId: 6bca23b2b3a2cfda","path: /folders/folders"],"remediation":null,"links":[ ],"version":2,"httpStatusCode":401} + +To fix the problem generate a new access token /opt/sas/viya/home/bin/sas-admin auth login + +To see the current setup, including python and package versions, environment variable settings, profile information and current user. Run + +*python showsetup.py* + +If you get this error : Login failed. Error with security certificate. + +Set the environment variable for the SSL certificate file. For example: +export SSL_CERT_FILE=/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem + + +If you get this error: +Raise SSLError(e, request=request) +requests.exceptions.SSLError: HTTPSConnectionPool(host='intviya01.race.sas.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, u'[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)'),)) + +Set the environment variable for the SSL certificate file. For example: +export REQUESTS_CA_BUNDLE=/opt/sas/viya/config/etc/SASSecurityCertificateFramework/cacerts/trustedcerts.pem + + +## Developing with the existing functions + +The file sharedfunctions.py contains a set of generic functions that make it easy to build additional tools. + +*callrestapi* is the main function, called by many other programs and by the callrestapi program to make the REST calls. It accepts as parameters: + +* reqval: the request value which is the rest endpoint for the request +* reqtype: the type of REST request, get,put, post, delete +* acceptType: optinal accept type content header +* contentType: optional content type content header +* data: optionally a python dictionary created from the json for the rest request +* stoponerror: whether the function will stop all further processing if an error occurs (default 0 to not stop) + + +## Contributing + +We welcome your contributions! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to submit contributions to this project. + +## License + +This project is licensed under the [Apache 2.0 License](LICENSE). + diff --git a/SUPPORT.md b/SUPPORT.md new file mode 100644 index 0000000..9e0179b --- /dev/null +++ b/SUPPORT.md @@ -0,0 +1,3 @@ +## Support + +We use GitHub for tracking bugs and feature requests. Please submit a GitHub issue or pull request for support. \ No newline at end of file