I’m also one but I don’t work for advertising. Most data engineers work for consulting companies that work for banks. We program automatic data processing pipelines. For example, bank transactions are stored somewhere, all the historic data, that needs processing to then be graphed out for exec number 3, or for whatever.
Other companies might send you files that need to be automatically processed, cleaned, and put correctly where then other tools can pull that data correctly.
We basically do all the background work concerning data manipulation. File processing, databases… all that stuff. And by databases it can be normal ones like posture to distributed ones like hdfs/hive/athena/whatever.
Ad world is basically the same but with tracking info instead of transactions.
If you are interested in day to day work, it’s a mix of coding SQL processes, then porting them to spark/pyspark for distributed massive processing. There are new shiny tools for those that don’t know much of the technical side to manage, sorta.
There are petabytes of collected data (even in a relatively small ad-tech companies I had a chance to work on; on a facebook/google scale it is much more). Someone should write all the cleaning, processing, de-duplication and matching (aka fingerprinting) steps as well as make this data usable by AI / Machine Learning guys, who will make models that predicts what ad to show to each user based on the available data. I’m working on processing, cleaning, matching and preparing of these data.
I know that I’m working on a “dark side”… But ad-tech are offering really interesting tasks about building very complex infrastructure besides they are paying well.
Oh, I never seen anyone say they worked as a data manager. Never even heard about this job.
How’s it? What do you do?
*data engineer
I’m also one but I don’t work for advertising. Most data engineers work for consulting companies that work for banks. We program automatic data processing pipelines. For example, bank transactions are stored somewhere, all the historic data, that needs processing to then be graphed out for exec number 3, or for whatever.
Other companies might send you files that need to be automatically processed, cleaned, and put correctly where then other tools can pull that data correctly.
We basically do all the background work concerning data manipulation. File processing, databases… all that stuff. And by databases it can be normal ones like posture to distributed ones like hdfs/hive/athena/whatever.
Ad world is basically the same but with tracking info instead of transactions.
If you are interested in day to day work, it’s a mix of coding SQL processes, then porting them to spark/pyspark for distributed massive processing. There are new shiny tools for those that don’t know much of the technical side to manage, sorta.
There are petabytes of collected data (even in a relatively small ad-tech companies I had a chance to work on; on a facebook/google scale it is much more). Someone should write all the cleaning, processing, de-duplication and matching (aka fingerprinting) steps as well as make this data usable by AI / Machine Learning guys, who will make models that predicts what ad to show to each user based on the available data. I’m working on processing, cleaning, matching and preparing of these data.
oooh
I know that I’m working on a “dark side”… But ad-tech are offering really interesting tasks about building very complex infrastructure besides they are paying well.
Another sem 👽