In all honesty, I feel like proper database solutions are just not as accessible to laypeople as they’d need to be. It’s easy to create a table in Excel, enter arbitrary values and share it. It’s also not particularly hard to create a second table and add some simple formula for a lookup. More complicated logic can be learned as you go.
By comparison, something like, say, Access needs more effort and understanding of data structures. You can eyeball a spreadsheet and just enter values without worrying about types, data integrity or anything. Never mind setting up actual database servers.
Yes, obviously those “proper” definitions would be more reliable, but particularly when the use cases aren’t entirely clear from the outset and new ones keep getting tacked on to an existing solution, it’s just more convenient in the moment to use a fairly low-effort solution until the whole thing becomes a clusterfuck of “low-effort” solutions.
It becomes a matter of platform gravity: By now, so many people are used to Excel and so much infrastructure is built around it that even a new, better and more laypeople-friendly data handling tool would have a hard time getting a foot in that door.
I consider myself pretty knowledgeable with most computing tasks, not particularly great with basic spreadsheets, but unless there’s some kind of usable frontend to reliably manage a database, I mostly see databases as:
“A magic box that holds tons of cryptic information, would be tedious to open, risky to edit, risky to backup or migrate or update, and could corrupt at any moment.”
Maybe I should put more effort into learning DBs besides initializing them in a Docker compose and praying, but for human readable information that’s meant to be shared, I think you’re bang on the money when it comes to why spreadsheets are still so popular!
Your view of databases is indicative of a different issue I tend to rag on about. This is by no means intended to discredit your feelings about it – they’re perfectly valid and understandable given your own background.
The issue I mean is that many things I consider easy to understand (with the caveat that my own familiarity will heavily skew what I consider “easy”) or at least basic useful or important knowledge are often misunderstood.
This should be a fairly simple fix with some superficial survey style trainings, which obviously requires both the time investment and the willingness to engage with the matter, neither of which is quite as simple as I’d like it to be. The biggest hurdle, in my experience, is finding a good teacher that can effectively simplify and communicate that surface-level overview.
To pick some examples from your comment, assuming we’re talking about SQL databases:
First off, you’re correct that, without training or expertise, opening and interacting with databases can be tedious. Some of this can and should be simplified (keywords: views, procedures built by more experienced users), and there are graphical tools for many database types too. Even with all that, it’ll never be quite as “trivial” as spreadsheets.
The data shouldn’t be cryptic. You define tables, much like you would in excel, except you have to be more explicit about their contents – Numbers? Text? Dates? Money? If you give your tables and columns reasonable names, this shouldn’t be too obscure.
You’re correct that, without some frontend providing guardrails, it’s easy to screw things up when editing or deleting data. I also think it’s easily avoidable with some simple habits: Before you update or delete, run a SELECT query with the same filter criteria to make sure you’ve only got those entries you actually want to edit.
You can also use transactions to “prepare” the change and only actually “save” them if it worked correctly. That’s three extra commands, with slight differences based on your SQL dialect, but the general outline is the same: Begin / Start your transaction, do your insert / delete / update, select to make sure the result is what you intended, then either commit (if it’s correct) or rollback (ig it’s not).
Modern databases should be resilient against corruption. Backups are usually simple, built-in functions. Migrations are often possible through simple wizards.
None of this will make databases simpler than (or even just as simple as) spreadsheets. The reason you’ll see experts advocate for them anyway is that the added effort (ideally) is rewarded with more reliability, more powerful functions and better interoperability with other tools.
As someone working with (specifically, reading and processing) structured data, databases generally provide some degree of clarity that Excel sheets don’t. I’ll be ever so grateful for you if you do use them.
But I can’t begrudge anyone for choosing the easier (in the short term) way.
As someone who interacts with databases regularly… Yeah, that sounds about right.
I was recently working with another team’s feature to handle data retrieval for the end user, pretty front end but it was far too tightly coupled with db management concepts. How is a non-technical person supposed to know the difference between an inner join and a left join?
Not too long ago I suggested using cross apply to a senior dev I work with and they admitted they weren’t sure what that does or how to use it. People who don’t regularly work with databases have no chance.
And then OneDrive comes along, someone accidentally saved “to the cloud” (IE the default windows location of OneDrive). And of course someone (you) has to fix all the desync bullshit.
Fuck excel, fuck Microsoft, fuck OneDrive!
Thank god my company is transitioning to a decent no code solution (nocobase plus literally anything that can interact with postgres - currently n8n but not yet limited to that. It’s a transition from excel, literally anything is better! (Tho, nocobase is awesome, non has it’s perks)).
Many parentheses, soz.
Fuck excel, use a database!
I’ve seen that. Used for customer service history AND planning with 3-digit week numbers (the first digit is the last digit of the year) and a lot of macros. Guess who had to fix the macros in 2020 without changing the idiotic 3-digit week numbers?
I’ve seen at a very large company a workflow that involved manually updating an excel workbook and (I think) saving it on confluence, so a python script could download it and parse it later. It wasn’t even doing formulas. It was just like less than a hundred lines of text in a half dozen sheets.
Professionals do seem to use excel.
Holy fuck is it painful for anyone that knows what they are doing.
In all honesty, I feel like proper database solutions are just not as accessible to laypeople as they’d need to be. It’s easy to create a table in Excel, enter arbitrary values and share it. It’s also not particularly hard to create a second table and add some simple formula for a lookup. More complicated logic can be learned as you go.
By comparison, something like, say, Access needs more effort and understanding of data structures. You can eyeball a spreadsheet and just enter values without worrying about types, data integrity or anything. Never mind setting up actual database servers.
Yes, obviously those “proper” definitions would be more reliable, but particularly when the use cases aren’t entirely clear from the outset and new ones keep getting tacked on to an existing solution, it’s just more convenient in the moment to use a fairly low-effort solution until the whole thing becomes a clusterfuck of “low-effort” solutions.
It becomes a matter of platform gravity: By now, so many people are used to Excel and so much infrastructure is built around it that even a new, better and more laypeople-friendly data handling tool would have a hard time getting a foot in that door.
I consider myself pretty knowledgeable with most computing tasks, not particularly great with basic spreadsheets, but unless there’s some kind of usable frontend to reliably manage a database, I mostly see databases as:
“A magic box that holds tons of cryptic information, would be tedious to open, risky to edit, risky to backup or migrate or update, and could corrupt at any moment.”
Maybe I should put more effort into learning DBs besides initializing them in a Docker compose and praying, but for human readable information that’s meant to be shared, I think you’re bang on the money when it comes to why spreadsheets are still so popular!
Your view of databases is indicative of a different issue I tend to rag on about. This is by no means intended to discredit your feelings about it – they’re perfectly valid and understandable given your own background.
The issue I mean is that many things I consider easy to understand (with the caveat that my own familiarity will heavily skew what I consider “easy”) or at least basic useful or important knowledge are often misunderstood.
This should be a fairly simple fix with some superficial survey style trainings, which obviously requires both the time investment and the willingness to engage with the matter, neither of which is quite as simple as I’d like it to be. The biggest hurdle, in my experience, is finding a good teacher that can effectively simplify and communicate that surface-level overview.
To pick some examples from your comment, assuming we’re talking about SQL databases:
First off, you’re correct that, without training or expertise, opening and interacting with databases can be tedious. Some of this can and should be simplified (keywords: views, procedures built by more experienced users), and there are graphical tools for many database types too. Even with all that, it’ll never be quite as “trivial” as spreadsheets.
The data shouldn’t be cryptic. You define tables, much like you would in excel, except you have to be more explicit about their contents – Numbers? Text? Dates? Money? If you give your tables and columns reasonable names, this shouldn’t be too obscure.
You’re correct that, without some frontend providing guardrails, it’s easy to screw things up when editing or deleting data. I also think it’s easily avoidable with some simple habits: Before you update or delete, run a
SELECTquery with the same filter criteria to make sure you’ve only got those entries you actually want to edit.You can also use transactions to “prepare” the change and only actually “save” them if it worked correctly. That’s three extra commands, with slight differences based on your SQL dialect, but the general outline is the same: Begin / Start your transaction, do your insert / delete / update, select to make sure the result is what you intended, then either commit (if it’s correct) or rollback (ig it’s not).
Modern databases should be resilient against corruption. Backups are usually simple, built-in functions. Migrations are often possible through simple wizards.
None of this will make databases simpler than (or even just as simple as) spreadsheets. The reason you’ll see experts advocate for them anyway is that the added effort (ideally) is rewarded with more reliability, more powerful functions and better interoperability with other tools.
As someone working with (specifically, reading and processing) structured data, databases generally provide some degree of clarity that Excel sheets don’t. I’ll be ever so grateful for you if you do use them.
But I can’t begrudge anyone for choosing the easier (in the short term) way.
As someone who interacts with databases regularly… Yeah, that sounds about right.
I was recently working with another team’s feature to handle data retrieval for the end user, pretty front end but it was far too tightly coupled with db management concepts. How is a non-technical person supposed to know the difference between an inner join and a left join?
Not too long ago I suggested using cross apply to a senior dev I work with and they admitted they weren’t sure what that does or how to use it. People who don’t regularly work with databases have no chance.
15000 rows. 120 columns. One sheet. Creation date: 2011. A dedicated computer. Working at a multinational company is bad for mental health.
And then OneDrive comes along, someone accidentally saved “to the cloud” (IE the default windows location of OneDrive). And of course someone (you) has to fix all the desync bullshit.
Fuck excel, fuck Microsoft, fuck OneDrive!
Thank god my company is transitioning to a decent no code solution (nocobase plus literally anything that can interact with postgres - currently n8n but not yet limited to that. It’s a transition from excel, literally anything is better! (Tho, nocobase is awesome, non has it’s perks)).
Many parentheses, soz.
Fuck excel, use a database!
I’ve seen that. Used for customer service history AND planning with 3-digit week numbers (the first digit is the last digit of the year) and a lot of macros. Guess who had to fix the macros in 2020 without changing the idiotic 3-digit week numbers?
FiServ. Distributed to clients.
I’ve seen at a very large company a workflow that involved manually updating an excel workbook and (I think) saving it on confluence, so a python script could download it and parse it later. It wasn’t even doing formulas. It was just like less than a hundred lines of text in a half dozen sheets.