Jeffrey Silverman

We're facing more and more questions about the use of synthetic data in market research. I'd recommend this blog post, which really resonated with me. So much of the conversation is "how closely can this replicate human data?" But there are so many other considerations researchers should be thinking about. https://lnkd.in/eSQfSTPj

31

3 Comments

Unlock powerful insights with DSS Professional's Intelligent Analysis. Metadata tools for demographic and directional data, including gender and age. 'In Area Number' analysis with custom duration settings for strategic planning. Download & learn more: http://tiny.cc/dahuadss

11

1 Comment

Just read a fun new post from Benjamin Rogojan (Seattle Data Guy) on why data teams shouldn’t blindly sprint toward Apache Iceberg. As always, Ben’s writing is provocative in the best way - challenging us to pause and ask why we’re reaching for the latest tech before we commit. I'm going to counter Ben on this one: Firsthand, I’d argue that there is a clear place for it. One where it not only accelerates time-to-insight, but also helps reduce the need to pile more junk onto the dump. In the analytics world, we’re constantly under pressure to answer questions fast. Stakeholders needed insights yesterday, so we chase whatever gets us to a solution quickest. Most of the time, those swift moves leave behind a heap of tech debt to clean up next year. But is Iceberg just another patch for deeper infrastructure issues …or does it actually have a practical home in your stack? Take 6sense: in 2022 calendar year alone, we went from two distinct warehouse environments to 5+. Why? M&A of 4 product companies and numerous new feature launches with evolving architecture needs.... But more environments don’t change expectations. Everyone still wants centralized analytics. A real question just two weeks after acquisition of company #3: “Can we just see how usage of feature X in warehouse A + adoption of capability Y in environment B impacts customer health stored in warehouse C?” Sounds easy, right? Just write a cross-environment query and wrap it up before lunch? In reality, you’re scrambling to stitch everything together, suddenly managing dozens (or hundreds) of EL jobs just to centralize data. At our peak, we had over 100 of these running nightly - each one draining compute, delaying other analytics pipelines, and straining SLA expectations. Would Iceberg have solved all of that? No. Integrating acquired systems and legacy stacks is never simple. But had we adopted Iceberg earlier - with strong governance and a cross-functional design approach that prioritized analytics from the start - we could’ve avoided a lot of duplication and orchestration overhead. Iceberg is NOT a silver bullet. But in the right context, it’s a powerful architectural lever for data democratization and it can absolutely drive both efficiency and cost savings. Curious to hear from others: Has your team adopted Iceberg? What made it worth it…or not? Highly recommend giving Ben a follow if you work in data. The post in reference in comments!

29

19 Comments

If your datasets are poorly labeled, you’re inviting a host of issues like noise, bias, and inconsistencies that can seriously hurt your model's accuracy and reliability when it’s out in the wild. That's why data labeling has evolved from being just a preliminary step into a vital part of engineering and operations in the AI lifecycle.

7

🎧 IDinsight’s Senior Statistician Douglas Johnson is on the latest episode of SurveyCTO’s Survey & Beyond podcast! In this episode, Doug joins host Marta Costa to unpack how IDinsight is leveraging cutting-edge technology to transform survey sampling, especially in contexts where traditional methods and census data fall short. From rooftop and satellite imagery to grid-based approaches, Doug shares our latest sampling innovations that balance rigor with cost-effectiveness to collect more reliable, inclusive data. 🔊 Shoutout to SurveyCTO for spotlighting innovations shaping the future of data collection. 🎙️ Start listening here: https://lnkd.in/ea2F_mcp 📖 PS: Don’t miss Doug’s latest blog, ‘Why survey data should be here to stay’: https://lnkd.in/guu2gGM6

42

"What if behavioral scientists could identify the relief of the behavioral landscape by working like cartographers?" Dilip Soman asks in his latest piece for Behavioral Scientist. Read it here: https://lnkd.in/eMBiTcmn

2

Census API (including those mediated by {tidycensus} ) may continue running through the federal government shutdown. It is unclear which services will be available.

7

1 Comment

Geospatial datasets are getting larger every year—and the file format you choose can have a big impact on storage, speed, and downstream analysis. In this post, Jared Lander and Joseph Marlo compare common formats like CSV, GeoJSON, and KML, explain why GeoParquet is becoming the preferred option for large datasets, and discuss when it’s time to move from files to databases like PostGIS. Read the full post: https://lnkd.in/epXkxhR6 #Geospatial #DataEngineering #DataScience

8