When it comes to solving a problem, developers often tend to use the technologies they’re comfortable with — the technologies they work with on a daily basis. Most of the time this is fine — we work within a specific scope in which this technology makes sense and has already proven relevant and efficient.
Although you might focus on one or a few specific technologies, in a software company you’re likely to work with other people tackling other subjects with different tools.
With an ever growing quantity of data to ingest and a limited pool of back-end machines, this started to become an issue.
Most of the content in these files is already in our databases, so on a day-to-day basis the ratio of new lines, updated lines, and deleted lines is rather small.