jenkins-bot has submitted this change and it was merged. (
https://gerrit.wikimedia.org/r/365543 )
Change subject: Update Omnimail GET to add rml fields
......................................................................
Update Omnimail GET to add rml fields
I think these fields are specific to our DB so have treated them like custom
data,
rather than adding getLanguage() etc functions.
Add Omnimmembergroup load functions
Perhaps I'm just Silverpopped out but this felt a bit harder to keep generic &
keep the lines between what should do what. I've cut a few corners, trying to
do more
wmf-like handling in the main load function. Hopefully I've mitigated that
somewhat
with tests & comments.
This should allow us to get our group of contacts with no id & to create into
our DB with
drush cvapi 'Omnigroupmember', 'load', array('mail_provider' => 'Silverpop',
'group_identifier' => 18468760, 'group_id' => 777);
Whre group_identifier is the id in the phab & group_id is the id of an
as-yet-to-be-created
group in CiviCRM.
Data brought in is
- email
- language (if we can figure out what it is)
- country (if matching DB countries)
- source is a combo of the fact it is silverpop + the source in the rml form +
the submit date
I thought about putting submit date in created_date but it's arguable whether
that is a good idea as there are 2 separate events. That can't be done through
the api on create if we do wish to.
- is_opt_out - see notes below
Is OPT out
in the data set I got some contacts were already opted out. I need to dig a
little
more here as there is opt in and opt out and I'm not sure if opt in is the
absence of opt
out or an active opt in. There are some flags that can be set when fetching
data that
can manipulate what is fetched to I need to answer that here.
Also, although I was purposely not setting the 'skip ones already downloaded'
some of
my later downloads seemed a little light - I need to do a little more testing
to check that
out.
I think clarifying these opt-out questions is probably the last step in the
silverpop chunk of
work
Bug T160949
Change-Id: Ia0dc97b84603ca3cd32aba17094593cfad95900f
---
M composer.lock
M
sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Get.php
A
sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Load.php
M
sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberGetTest.php
A
sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberLoadTest.php
M
sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/Responses/20170509_noCID
- All - Jul 5 2017 06-27-45 AM.csv
6 files changed, 252 insertions(+), 4 deletions(-)
Approvals:
jenkins-bot: Verified
Ejegg: Looks good to me, approved
diff --git a/composer.lock b/composer.lock
index eb004f2..b6be6ff 100644
--- a/composer.lock
+++ b/composer.lock
@@ -1842,7 +1842,7 @@
"source": {
"type": "git",
"url":
"https://github.com/eileenmcnaughton/omnimail-silverpop.git";,
- "reference": "c48de4450ec495a8fd1cb26ba5e656876321e025"
+ "reference": "cd3933b8ee7b263d0472cc44d047b82f3ec4029c"
},
"require": {
"league/csv": "^8.0",
diff --git
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Get.php
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Get.php
index e813920..0f179f6 100644
---
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Get.php
+++
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Get.php
@@ -19,7 +19,7 @@
$options = _civicrm_api3_get_options_from_params($params);
$values = array();
foreach ($result as $groupMember) {
- $values[] = array(
+ $value = array(
'email' => (string) $groupMember->getEmail(),
'is_opt_out' => (string) $groupMember->isOptOut(),
'opt_in_date' => (string) $groupMember->getOptInIsoDateTime(),
@@ -28,6 +28,10 @@
'opt_out_date' => (string) $groupMember->getOptOutIsoDateTime(),
'contact_id' => (string) $groupMember->getContactReference(),
);
+ foreach ($params['custom_data_map'] as $fieldName => $dataKey) {
+ $value[$fieldName] = (string) $groupMember->getCustomData($dataKey);
+ }
+ $values[] = $value;
if ($options['limit'] > 0 && count($values) === (int) $options['limit']) {
break;
}
@@ -68,5 +72,16 @@
$params['retrieval_parameters'] = array(
'title' => ts('Additional information for retrieval of pre-stored
requests'),
);
+ $params['custom_data_map'] = array(
+ 'type' => CRM_Utils_Type::T_STRING,
+ 'title' => ts('Custom fields map'),
+ 'description' => array('custom mappings pertaining to the mail provider
fields'),
+ 'api.default' => array(
+ 'language' => 'rml_language',
+ 'source' => 'rml_source',
+ 'created_date' => 'rml_submitdate',
+ 'country' => 'rml_country',
+ ),
+ );
}
diff --git
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Load.php
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Load.php
new file mode 100644
index 0000000..5fcc26f
--- /dev/null
+++
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/api/v3/Omnigroupmember/Load.php
@@ -0,0 +1,145 @@
+<?php
+/**
+ * Created by IntelliJ IDEA.
+ * User: emcnaughton
+ * Date: 5/3/17
+ * Time: 12:46 PM
+ */
+
+/**
+ * Get details about Omnimails.
+ *
+ * @param $params
+ *
+ * @return array
+ */
+function civicrm_api3_omnigroupmember_load($params) {
+ $contacts = civicrm_api3('Omnigroupmember', 'get', $params);
+ $defaultLocationType = CRM_Core_BAO_LocationType::getDefault();
+ $locationTypeID = $defaultLocationType->id;
+
+ foreach ($contacts['values'] as $groupMember) {
+ if (!empty($groupMember['email']) && !civicrm_api3('email', 'getcount',
array('email' => $groupMember['email']))) {
+ // If there is already a contact with this email we will skip for now.
+ // It might that we want to create duplicates, update contacts or do
other actions later
+ // but let's re-assess when we see that happening. Spot checks only
found emails not
+ // otherwise in the DB.
+ $source = (empty($params['mail_provider']) ? ts('Mail Provider') :
$params['mail_provider']) . ' ' . (!empty($groupMember['source']) ?
$groupMember['source'] : $groupMember['opt_in_source']);
+ $source .= ' ' . $groupMember['created_date'];
+
+ $contactParams = array(
+ 'contact_type' => 'Individual',
+ 'email' => $groupMember['email'],
+ 'is_opt_out' => $groupMember['is_opt_out'],
+ 'source' => $source,
+ 'preferred_language' =>
_civicrm_api3_omnigroupmember_get_language($groupMember),
+ );
+
+ if (!empty($groupMember['country']) &&
_civicrm_api3_omnigroupmember_is_country_valid($groupMember['country'])) {
+ $contactParams['api.address.create'] = array(
+ 'country_id' => $groupMember['country'],
+ 'location_type_id' => $locationTypeID,
+ );
+ }
+
+ $contact = civicrm_api3('Contact', 'create', $contactParams);
+ if (!empty($params['group_id'])) {
+ civicrm_api3('GroupContact', 'create', array(
+ 'group_id' => $params['group_id'],
+ 'contact_id' => $contact['id'],
+ ));
+ }
+ $values[$contact['id']] = reset($contact['values']);
+ }
+ }
+ return civicrm_api3_create_success($values);
+}
+
+/**
+ * Get the contact's language.
+ *
+ * This is a place in the code where I am struggling to keep wmf-specific
coding out
+ * of a generic extension. The wmf-way would be to call the wmf contact_insert
function.
+ *
+ * That is not so appropriate from an extension, but we have language/country
data that
+ * needs some wmf specific handling as it might or might not add up to a legit
language.
+ *
+ * At this stage I'm compromising on containing the handling within the
extension,
+ * ensuring test covering and splitting out & documenting the path taken
/issue.
+ * Later maybe a more listener/hook type approach is the go.
+ *
+ * It's worth noting this is probably the least important part of the omnimail
work
+ * from wmf POV.
+ *
+ * @param array $params
+ *
+ * @return string|null
+ */
+function _civicrm_api3_omnigroupmember_get_language($params) {
+ static $languages = NULL;
+ if (!$languages) {
+ $languages = civicrm_api3('Contact', 'getoptions', array('field' =>
'preferred_language', 'limit' => 0));
+ $languages = $languages['values'];
+ }
+ $attempts = array(
+ $params['language'] . '_' . strtoupper($params['country']),
+ $params['language'],
+ );
+ foreach ($attempts as $attempt) {
+ if (isset($languages[$attempt])) {
+ return $attempt;
+ }
+ }
+ return NULL;
+}
+
+/**
+ * Check if the country is valid.
+ *
+ * @param string $country
+ *
+ * @return bool
+ */
+function _civicrm_api3_omnigroupmember_is_country_valid($country) {
+ static $countries = NULL;
+ if (!$countries) {
+ $countries = CRM_Core_PseudoConstant::countryIsoCode();
+ }
+ return array_search($country, $countries) ? $country : FALSE;
+}
+
+/**
+ * Get details about Omnimails.
+ *
+ * @param $params
+ */
+function _civicrm_api3_omnigroupmember_load_spec(&$params) {
+ $params['username'] = array(
+ 'title' => ts('User name'),
+ );
+ $params['password'] = array(
+ 'title' => ts('Password'),
+ );
+ $params['mail_provider'] = array(
+ 'title' => ts('Name of Mailer'),
+ 'api.required' => TRUE,
+ );
+ $params['start_date'] = array(
+ 'title' => ts('Date to fetch from'),
+ 'api.default' => '3 days ago',
+ 'type' => CRM_Utils_Type::T_TIMESTAMP,
+ );
+ $params['end_date'] = array(
+ 'title' => ts('Date to fetch to'),
+ 'type' => CRM_Utils_Type::T_TIMESTAMP,
+ );
+ $params['group_identifier'] = array(
+ 'title' => ts('Identifier for the group'),
+ 'type' => CRM_Utils_Type::T_STRING,
+ 'api.required' => TRUE,
+ );
+ $params['retrieval_parameters'] = array(
+ 'title' => ts('Additional information for retrieval of pre-stored
requests'),
+ );
+
+}
diff --git
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberGetTest.php
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberGetTest.php
index 086029c..0509181 100644
---
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberGetTest.php
+++
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberGetTest.php
@@ -52,7 +52,10 @@
$this->assertEquals('2017-07-04 11:11:00',
$result['values'][0]['opt_out_date']);
$this->assertEquals('Added by WebForms',
$result['values'][0]['opt_in_source']);
$this->assertEquals('Opt out via email opt out.',
$result['values'][0]['opt_out_source']);
-
+ $this->assertEquals('clever place', $result['values'][2]['source']);
+ $this->assertEquals('US', $result['values'][2]['country']);
+ $this->assertEquals('en', $result['values'][2]['language']);
+ $this->assertEquals('07/04/17', $result['values'][2]['created_date']);
}
diff --git
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberLoadTest.php
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberLoadTest.php
new file mode 100644
index 0000000..27c9c94
--- /dev/null
+++
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/OmnigroupmemberLoadTest.php
@@ -0,0 +1,85 @@
+<?php
+
+use Civi\Test\EndToEndInterface;
+use Civi\Test\HookInterface;
+use Civi\Test\TransactionalInterface;
+use GuzzleHttp\Client;
+use GuzzleHttp\Handler\MockHandler;
+use GuzzleHttp\HandlerStack;
+use GuzzleHttp\Psr7\Response;
+require_once __DIR__ . '/OmnimailBaseTestClass.php';
+
+/**
+ * FIXME - Add test description.
+ *
+ * Tips:
+ * - With HookInterface, you may implement CiviCRM hooks directly in the test
class.
+ * Simply create corresponding functions (e.g. "hook_civicrm_post(...)" or
similar).
+ * - With TransactionalInterface, any data changes made by setUp() or
test****() functions will
+ * rollback automatically -- as long as you don't manipulate schema or
truncate tables.
+ * If this test needs to manipulate schema or truncate tables, then either:
+ * a. Do all that using setupHeadless() and Civi\Test.
+ * b. Disable TransactionalInterface, and handle all setup/teardown
yourself.
+ *
+ * @group e2e
+ */
+class OmnigroupmemberLoadTest extends OmnimailBaseTestClass implements
EndToEndInterface, TransactionalInterface {
+
+ public function setUpHeadless() {
+ // Civi\Test has many helpers, like install(), uninstall(), sql(), and
sqlFile().
+ // See:
https://github.com/civicrm/org.civicrm.testapalooza/blob/master/civi-test.md
+ return \Civi\Test::e2e()
+ ->installMe(__DIR__)
+ ->apply();
+ }
+
+ public function tearDown() {
+ parent::tearDown();
+ }
+
+ /**
+ * Example: Test that a version is returned.
+ */
+ public function testOmnigroupmemberLoad() {
+ $client = $this->setupSuccessfulDownloadClient();
+ $group = civicrm_api3('Group', 'create', array('name' => 'Omnimailers',
'title' => 'Omni'));
+
+ civicrm_api3('Omnigroupmember', 'load', array('mail_provider' =>
'Silverpop', 'username' => 'Shrek', 'password' => 'Fiona', 'options' =>
array('limit' => 3), 'client' => $client, 'group_identifier' => 123, 'group_id'
=> $group['id']));
+ $groupMembers = civicrm_api3('GroupContact', 'get', array('group_id' =>
$group['id']));
+ $this->assertEquals(3, $groupMembers['count']);
+ $contactIDs = array('IN' => array());
+ foreach ($groupMembers['values'] as $groupMember) {
+ $contactIDs['IN'][] = $groupMember['contact_id'];
+ }
+ $contacts = civicrm_api3('Contact', 'get', array(
+ 'contact_id' => $contactIDs,
+ 'sequential' => 1,
+ 'return' => array('contact_source', 'email', 'country', 'created_date',
'preferred_language', 'is_opt_out')
+ ));
+ $this->assertEquals('fr_FR', $contacts['values'][0]['preferred_language']);
+ $this->assertEquals('e...@example.com', $contacts['values'][0]['email']);
+ $this->assertEquals('France', $contacts['values'][0]['country']);
+ $this->assertEquals(1, $contacts['values'][0]['is_opt_out']);
+ $this->assertEquals('Silverpop Added by WebForms 10/18/16',
$contacts['values'][0]['contact_source']);
+
+ $this->assertEquals('Silverpop clever place 07/04/17',
$contacts['values'][2]['contact_source']);
+
+ }
+
+
+ /**
+ * @return \GuzzleHttp\Client
+ */
+ protected function setupSuccessfulDownloadClient() {
+ $responses = array(
+ file_get_contents(__DIR__ . '/Responses/ExportListResponse.txt'),
+ file_get_contents(__DIR__ . '/Responses/JobStatusCompleteResponse.txt'),
+ );
+ copy(__DIR__ . '/Responses/20170509_noCID - All - Jul 5 2017 06-27-45
AM.csv', sys_get_temp_dir() . '/20170509_noCID - All - Jul 5 2017 06-27-45
AM.csv');
+ fopen(sys_get_temp_dir() . '/20170509_noCID - All - Jul 5 2017 06-27-45
AM.csv.complete', 'c');
+ $this->createSetting('omnimail_omnigroupmembers_load', array('Silverpop'
=> array('last_timestamp' => '1487890800')));
+
+ $client = $this->getMockRequest($responses);
+ return $client;
+ }
+}
diff --git
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/Responses/20170509_noCID
- All - Jul 5 2017 06-27-45 AM.csv
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/Responses/20170509_noCID
- All - Jul 5 2017 06-27-45 AM.csv
index 0f37c0d..b3e791d 100644
---
a/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/Responses/20170509_noCID
- All - Jul 5 2017 06-27-45 AM.csv
+++
b/sites/default/civicrm/extensions/org.wikimedia.omnimail/tests/phpunit/Responses/20170509_noCID
- All - Jul 5 2017 06-27-45 AM.csv
@@ -1,4 +1,4 @@
Email,Opt In Date,Opted Out,Opt In Details,Email Type,Opted Out Date,Opt Out
Details,ContactID,IsoLang,LastClickDate,LastOpenDate,LastSentDate,Segment,Segment_AOL,Segment_Newsletter,Segment_Newsletter2,SendHour,firstname,lastname,postal_code,rml_country,rml_device,rml_language,rml_phone,rml_segment,rml_source,rml_submitDate,state,timezone
e...@example.com,10/18/16 08:01 PM,T,Added by WebForms,0,07/04/17 11:11 AM,Opt
out via email opt out.,,,,12/04/16,10/22/16,0,0,0,0,9,,,,FR,,fr,,35,,10/18/16,,
sa...@example.com,12/13/16 12:27 AM,T,Added by WebForms,0,07/03/17 03:34
PM,Optout by WebForms,,,,,12/24/16,0,0,0,0,0,,,,US,,en,,83,,12/12/16,,
-l...@example.com,04/26/17 02:18 PM,F,Added by
WebForms,0,,,,,,,,0,0,0,0,0,,,,US,,qb,,28,clever place,07/04/17,,
+l...@example.com,04/26/17 02:18 PM,F,Added by
WebForms,0,,,,,,,,0,0,0,0,0,,,,US,,en,,28,clever place,07/04/17,,
--
To view, visit https://gerrit.wikimedia.org/r/365543
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ia0dc97b84603ca3cd32aba17094593cfad95900f
Gerrit-PatchSet: 5
Gerrit-Project: wikimedia/fundraising/crm
Gerrit-Branch: master
Gerrit-Owner: Eileen <emcnaugh...@wikimedia.org>
Gerrit-Reviewer: Cdentinger <cdentin...@wikimedia.org>
Gerrit-Reviewer: Eileen <emcnaugh...@wikimedia.org>
Gerrit-Reviewer: Ejegg <ej...@ejegg.com>
Gerrit-Reviewer: Mepps <me...@wikimedia.org>
Gerrit-Reviewer: jenkins-bot <>
_______________________________________________
MediaWiki-commits mailing list
MediaWiki-commits@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
