To tackle this uncertainty, we’ve enriched our optimization toolkit RevenueAccelerate, by using our machine learning intelligence to assess every transaction, beyond just exemptions. By analyzing our platform-wide data, the Authentication Engine can identify insights as the PSD2 landscape matures. Plus, at a bank level, the Authentication Engine monitors, identifies patterns and behaviors, and acts on them in real-time.
In order to connect these two different systems, we created Spidermon. Spidermon is a Java application that lives within the Adyen Payments Platform whose purpose is to retrieve real-time data and run algorithms and queries (i.e., monitors) to check whether what it is happening makes sense. Spidermon can be configured by operations since it offers a spectrum of monitors and parameters. Whenever Spidermon finds something odd, it raises a signal to Sonar, who enhances the context and routes it through a configured channel (e.g., SMS, email, Zendesk, our Back Office, etc.). The underperforming merchant is one of the monitors running in Spidermon.
PII data (this could be a name, email address, credit card details, etc) coming in from the left passes through our Tokenization system, which encrypts the data with a key from the HSM-backed vault key manager. The encrypted data is stored inside the Data Vault, and assigned a UUID that is unique for the context of the owner of the data, and the UUID becomes the token. The token then replaces the PII elements in the data, and stored in the Tokenized Data Lake, to be used by downstream processes.
This worked well for a period of time, until people actively started caching their dataframes. This means the dataframes are stored in the memory of the executor. By default, executors with cached data are not subject to deallocation, so each notebook would effectively stay at their maxExecutors, rendering dynamic (de)allocation useless. At that time this option was under-documented. After trial and error we figured out that within our organization, five minutes of inactivity is the most optimal period to shut off access to compute resources:
As a final note, sometimes in tech, people rush into adapting new technologies to “get things done” without considering the dangers. It’s the equivalent of a driver saying “We’re going really fast in this car, no roof, no brakes, no seatbelt, but we’re really getting places!”. Our aim is to help our developers to go far, go fast, and arrive at their destinations safely.