Product Academy
      Back to home
      1. Help & Support
      2. Product Academy

      Guide for SFTP integration

      Introduction

      This article provides a high-level summary of setting up SFTP integrations with Emitwise to programmatically send your activity data for emissions calculation.

      SFTP is a simple and secure way to automatically exchange files. We find it particularly successful with our customers using SAP.  

      Integration options

      1. On Premise to Cloud (File based)

      This integration pattern allows source system to create a data file (automatic or manual) and use the integration flow to securely transfer from on-premise to cloud target. 

      Suggested Integration Pattern Steps: 

      1. Extract data from the customer's sources
        1. Extract programs or routines are executed automatically in the source system and the “Data file” is created. Schedule is setup in the source system to extract the data (Monthly, Quarterly or Annually).
        2. “Data files” are placed in a specific folder within the on-premise staging file server. Access to this file server is restricted to authorised users of the source system or users of manual source. 
        3. The file is then encrypted using PGP Public key supplied by Emitwise before submitting it to the Emitwise SFTP Server. Authentication is using a SSH Keypair. 
      2. Emitwise receives files on SFTP
        1. Encrypted “Source Data files” are placed in a specific folder as defined with the customer
        2. Inbound processing at Emitwise will “Decrypt” the “Data files” using the PGP private key and the decrypted file is transferred for further processing.
        3. When the source system supplies the data files in non-Emitwise template, a pre-processing step is required at Emitwise to convert the incoming file format to Emitwise format. This step is only required when the source system supplied format is not in Emitwise template. 
        4. “Source Data file” in Emitwise template is processed and data written into Emitwise database for subsequent reporting or analytics. Data validations are carried out before the file is accepted, Any rejections are reported.
        5. At the end of the processing, Emitwise can trigger emails to the customer to confirm success.
        6. Processed files can be visible in the Collect section of the customer account

      2. On Premise to Cloud (Email based)

      This integration pattern extends on the first pattern and it allows source system or manual sources to email data files to a customer email account. An integration flow is used to securely transfer this data from on-premise to cloud target. 

      Suggested Integration Pattern Steps: 

      1. Extract data from the customer's sources
        1. Extract programs or routines are executed automatically in the source system and the “Data file” is created. Schedule is setup in the source system to extract the data (Monthly, Quarterly or Annually).
        2. “Data files” are manually created either from a manual source or from a source system where there is no native way of extracting the data automatically. This “Data file” is then emailed to a Customer email for processing. Schedule is managed manually. “Data files” are received in the Customer Mail Server.
        3. Interface flows connects to Customer Mail Server and pulls any new emails using SMTP protocol. 
        1. The file is then encrypted using PGP Public key supplied by Emitwise before submitting it to the Emitwise SFTP Server. Authentication is using a SSH Keypair. 

      An alternate way of sending source files to Emitwise is to send “Source Data Files” as email attachments to a pre-agreed Customer Email(tbc.email.id@customer.com). The emails to this id could be triggered from a source system in an automated way or from a data user who is manually collecting the required data. An integration layer connects to the Customer Email Server and pulls new emails using SMTP protocol. A separate “Integration flow” is required to further process the email attachments, encrypt and to send to Emitwise using SFTP connection.

      1. Emitwise receives files on SFTP
        1. Encrypted “Source Data files” are placed in a specific folder as defined with the customer
        2. Inbound processing at Emitwise will “Decrypt” the “Data files” using the PGP private key and the decrypted file is transferred for further processing.
        3. When the source system supplies the data files in non-Emitwise template, a pre-processing step is required at Emitwise to convert the incoming file format to Emitwise format. This step is only required when the source system supplied format is not in Emitwise template. 
        4. “Source Data file” in Emitwise template is processed and data written into Emitwise database for subsequent reporting or analytics. Data validations are carried out before the file is accepted, Any rejections are reported.
        5. At the end of the processing, Emitwise can trigger emails to the customer to confirm success.
        6. Processed files can be visible in the Collect section of the customer account

      3. Cloud to Cloud (File based)

      This integration pattern covers the Cloud-to-Cloud integration flow. Source Cloud applications have more than one way to integrate and one of these potential integration patterns is detailed below.

      Suggested Integration Pattern Steps: 

      1. Extract data from the customer's sources
        1. Extract programs or routines are executed automatically in the cloud source system, “Data file” is created and encrypted using PGP Public key supplied by Emitwise
        2. Source system’s STP Client is configured to connect with Emitwise SFTP Server and to transfer the encrypted “Data file”. Schedule is setup in the source system to extract the data (Monthly, Quarterly or Annually). Authentication is using a SSH Keypair. 
      2. Emitwise receives files on SFTP
        1. Encrypted “Source Data files” are placed in a specific folder as defined with the customer
        2. Inbound processing at Emitwise will “Decrypt” the “Data files” using the PGP private key and the decrypted file is transferred for further processing.
        3. When the source system supplies the data files in non-Emitwise template, a pre-processing step is required at Emitwise to convert the incoming file format to Emitwise format. This step is only required when the source system supplied format is not in Emitwise template. 
        4. “Source Data file” in Emitwise template is processed and data written into Emitwise database for subsequent reporting or analytics. Data validations are carried out before the file is accepted, Any rejections are reported.
        5. At the end of the processing, Emitwise can trigger emails to the customer to confirm success.
        6. Processed files can be visible in the Collect section of the customer account

      Integrations register

      For a successful integration, the first thing we suggest setting up is an integrations register. This is a spreadsheet mapping your source systems and source data fields to Emitwise's data schemas (data collection templates). 

      Emitwise can help you create this with your data owners and IT team. 

      Sharing access

      Emitwise will provide SFTP Server endpoint details and the User name to the Customer. Customer will need to create a SSH Keypair and share the public key with Emitwise. This SSH public key will be assigned to the Customer user at Emitwise to enable user authentication.

      Testing

      To test the integration, we recommend creating two separate folders for "Test" and "Production" interfaces. 

      Customer and Emitwise will first set up the integration to land on "Test". Once we've confirmed it's successfully working, the customer can change the target destination to the "Production" folder. 

      “Test” and “Production” systems will have different SFTP Endpoints, Users and SSH Key Pair.

      PGP Encryption

      Data classification for all data handled through this integration is confidential and thus should be encrypted both “in-Transit” and “at-Rest”. This is in addition to the protocol encryption of SFTP transfer using SSH.

      Emitwise will generate a PGP key pair for both “Test” and “Production” systems and share the public key with both the customer for encrypting “Source Data Files”.

      These PGP public keys will be shared with the customer as part of Build/Cutover phase.

      “Source Data files” are encrypted as it exits the customer landscape. These encrypted files are received at Emitwise and decrypted using the PGP private key in the SFTP folders.

      Monitoring

      Monitoring will be required to track the progress of each file from Source to Target. Since the file pass through various integration components, it is essential to have some kind of monitoring to pick up any failures.

      On the customer end, monitoring tools could be used to find the status of each system transfer, and the integration flows could be designed to trigger alerts when an exception is raised.

      On Emitwise, we can trigger an automated email to a customer email address when a new file is received for processing. Data files will also be displayed in the customer's account, under "Collect".