On Sat, Sep 5, 2015 at 4:22 AM, Ozgun Erdogan <oz...@citusdata.com> wrote:
> Hey Robert, > > Now the question is, where should the code that does all of this live? >> postgres_fdw? Some new, sharding-specific FDW? In core? I don't >> know for sure, but what I do know is that we could make a lot of >> progress over where we are today by just improving postgres_fdw, and I >> don't think those improvements are even all that difficult. If we >> decide we need to implement something new, it's going to be a huge >> project that will take years to complete, with uncertain results. I'd >> rather have a postgres_fdw-based implementation that is imperfect and >> can't handle some kinds of queries in 9.6 than a promise that by 9.9 >> we'll have something really great that handles MPP perfectly. >> > > Distributed shuffles (Map/Reduce) are hard. When we looked at using FDWs > for pg_shard, we thought that Map/Reduce would require a comprehensive > revamp of the APIs. > > For Citus, a second part of the question is as FDW writers. We implemented > cstore_fdw, json_fdw, and mongo_fdw, and these wrappers don't benefit from > even the simple join pushdown that doesn't require Map/Reduce. > I didn't get this. Join pushdown infrastructure (chiefly set of hooks provided in join planning paths) is part of 9.5. Isn't that sufficient to implement join push-down for above FDWs? Or FDW writers are facing problems while implementing those hooks. In either case that should be reported on hackers. > > The PostgreSQL wiki lists 85 foreign data wrappers, and only 18 of these > have support for joins: > https://wiki.postgresql.org/wiki/Foreign_data_wrappers > > Best, > Ozgun > -- Best Wishes, Ashutosh Bapat EnterpriseDB Corporation The Postgres Database Company
