Importing/Uploading Data in WP Issues CRM

WP Issues CRM provides a robust upload () facility:

  • It chunks processing tasks so as not to exceed memory or file size limits on smaller servers when handling large data sets.
  • It allows you to upload both activities and constituents simultaneously or separately.
  • It can create new issues for classifying activities as it goes.
  • It validates data transparently so that you can fix errors or bypass them.
  • It matches new constituent records with existing records using multiple possible match rules in a transparent way that you fully control.
  • It allows you to set defaults to complete partial incoming data (for example, for a list of signers of a petition, add them to the database and also create an activity record reflecting their signing of the petition).
  • In every instance it tells you clearly what the consequences of your taking the next upload step will be, and in most instances, it gives you the ability to roll-back any mistakes you make.
  • It offers a streamlined upload option to simplify the upload of clean new data.

It begins with the familiar operating system file selection dialog:uploadinit
It uploads the file in chunks showing a progress bar and then presents file parsing options — it handles most flavors of csv file:
Once the file has been parsed successfully, WP Issues CRM offers a friendly drag-and-drop interface for associating uploaded fields with existing database fields:
Once the user maps the data, the user can run validation. WP Issues CRM will apply validation rules appropriate to the fields as mapped:uploadvalidate
For much more on how WP Issues CRM validates incoming records, please see this post.

Continue to read: The next step in the Upload Process — Matching.


WP Issues CRM is built around three simple data constructs:  Constituents, Issues and Activities.


Constituents are people (or institutions) and you can save basic information about them, including whatever custom information you might choose to define through custom fields.  WP Issues CRM stores constituents in custom tables within the WordPress database for fast retrieval.



Issues can be used to represent anything that a constituent or constituent activity might have a relationship to.  An issue could be a public policy issue like “Charter Schools” and you could log incoming email as pertaining to that issue.  Or, an issue could be “the Town Club” and you could identify constituents as members.   Or an issue could be the 2016 Presidential Primary and you could record people as having voted in that election.  WP Issues CRM stores issues as WordPress posts.  WP Issues CRM stores the issue posts as private, but you can make them public or use any public post from your website as an issue.saveissue


Activities can be used to represent anything that a constituent might do or be with respect to an issue.  You can define whatever set of activity types you wish to.  You could have “Email”, “Letter”, and “Call” as activity types and use them to record incoming contacts from constituents and classify them by issue.  Or you could define activities like “Is a Member of” or “Voted in”.  saveactivity
The activity add form is available from within the issue and constituent forms.

Classifying and retrieving constituents and activities

WP Issues CRM offers multiple approaches to classifying and retrieving constituents and activities.

Most importantly, you have all the flexibility that WordPress offers to tag and categorize posts. The advanced search shown below would retrieve all constituents with a home address in Boston who had sent two or more emails about any issue classified under any sub-category under the environment category since January 1, 2016.advanced-search

Additionally, you can create custom fields for constituents, including drop down fields with option sets that you define. For example, you could add a “Party” field and classify voters by party.  (The party field is built-in in recent versions of WP Issues CRM).


Finally, you can create custom activity types that fit the activity of your office. Activities could be relationships like “is a member of” or “subscribed to”.options

Tracking financial activities

WP Issues CRM supports financial activity tracking. You could use it, for example, to track contributions. You could then retrieve and download contribution information with all the same flexibility that you have for other activities.

To show amounts on all reports and forms for a particular activity type is identify they type as financial on the Settings form:

To protect the integrity of your bookkeeping after a closing date, you can freeze transactions by date, again from the settings screen:

Issues and Activities in the Upload Process

As explained here, the upload process can support the addition of constituent and activity records together or separately.

In the simplest case, one is uploading a file of constituents with no activity data explicitly in it and appending a default activity record at the default setting stage — for example, if all of the constituents uploaded were signers of a petition.uploaddefault

It is possible, however, to supply activity data in the input file itself.  In that case, each activity record in the file needs to identify the issue that it is associated with.  You can do this two ways — either by mapping an input column to issue number (which is the WordPress post ID of the issue, visible in the URL for editing the issue) OR by specifying the exact title of the issue.  If you map issue number, it has to be a valid existing number.   If you map title, however, WP Issues CRM will, at the default stage, offer you the option of creating new issues out of unmatched titles.  The content of the created issue is unpredictable in a file with multiple identical title lines.

Advanced Search Tips

A few concepts to bear in mind:

  1. Most of the time you’ll find the individual constituent or issue you need by using the quick search box in the top bar — you only need advanced search to retrieve a complex set of constituents or activities.
  2. When you do an advanced search with no search terms set, you are retrieving a table of all the constituents and all their activities.
  3. The search terms that you add exclude activities and/or constituents from that table.
  4. There are three kinds of search terms that you can specify:
    • Constituent Search Terms
    • Activity Search Terms
    • Constituent Activity Search Terms
  5. If you define the search to retrieve constituents:
    • Obviously, the list will only include constituents meeting the Constituent Search terms you have chosen.
    • But also, the list will only include constituents that have activities meeting the Activity Search Terms you entered.
    • The Constituent Activity Search Terms are applied for each constituent to their set of activities — the Constituent Activity Search terms are applied after the Constituent Search Terms and the Activity Search Terms.
  6. Notice that you have options for how you combine the search terms:
    • Require all
    • Require any
    • Require all false
    • Require any false
  7. For both Activity Search Terms and Constituent Activity Search Terms, you can use Issue category selections. These will be applied first to define a set of issues applicable only to that Search Term row. For Constituent Activity Search Terms, the single way of combining multiple issue categories is to retrieve all selected and all of their descendants.
  8. For Constituent Activity Search Terms, the left hand operator defines what the constituent selection is — constituents having Sum, Max, Count etc. meeting the right hand comparison. The middle terms, type and category selections, further limit the activity records included in the Sum or Max or Count from within the set already defined by the Activity Search Terms and Constituent Search Terms. See the examples.

Some Examples:


Problem: How would I find everyone who had ever contacted me on an environmental issue and had sent me at least one email on any subject in the last six months — in other words, they care about environmental issues and I have a fresh email address for them?

Solution: Choose to Retrieve constituents in the Search Definition and add two Constituent Activity Search Terms:

  1. Maximum — Date — Email — No Cat Selection — Greater than or equal to — 2015-03-01
  2. Count — Date — No Type Selection — Cat Environmental — Greater than or equal to — 0

Note that in the second search term ActivityDate is not tested per se — any selected field produces the same result if one is just counting records, because no fields on the Activity Record are allowed to have “Null” values; they are all countable.


Problem: How can I list all the constituents for whom I’ve added an email address since July 1, 2015?

Solution: Choose to Retrieve constituents in the Search Definition and add a single Constituent Search Term:

  • Update Time — Email — Greater than or equal to — 2015-07-01

You would then be able to export the constituents and their emails from the constituent list.


Problem: How would I list all the checks and online contributions I’ve received since July 1, 2015?

Solution: Choose to Retrieve activities in the Search Definition and add two Activity Search Terms:

  1. Date — Check — Greater than or equal to — 2015-07-01
  2. Date — Online Contribution — Greater than or equal to — 2015-07-01
  3. Don’t forget to select “Require Any” — if you require all, you are requiring your activities to be both checks and online contributions and that will yield no records.

You could add Constituent Search Terms, for example, selecting a city to narrow the list to include only contributions from that City.

This presumes that you have gone to Options and set up some appropriate activity types (e.g., check and online contribution) and have gone to Settings and have designated them as financial. With those parameters in place, you can retrieve contributions as above. No financial activity types are configured by default for WP Issues CRM.


  1. I selected activities with last update time equal to today. I found nothing. What gives? Use greater than or equal. The time stamp actually includes the minute and second of the update, but that’s not something you will typically know. Retrieving last update time greater than or equal to 2015-03-01 will retrieve all the time stamps on that date and after it.

The Match Process in WP Issues CRM Uploads

WP Issues CRM gives you full control over whether uploaded constituents will be considered to be new constituents or to match existing constituents in your database. You control when a record on your input file will be treated as being about a person already in the system. Is it enough that they have the same first and last name? Or do you also want to require date of birth to be identical? Or also require street address?

If records match, say on first name and last name, then the data in your input file will be used to update the record on the database. If a record on the input file matches no one on the database, then it will be added as a new constituent record. ( You can override these decisions in settings in the default setting step.)

The basic philosophy of WP Issues CRM is that you should try to match as many as possible using strict criteria and then use looser criteria to match out others for whom you may have incomplete or imperfect data. WP Issues CRM attempts matches between the upload file and your existing database records in multiple passes in the order that you prioritize the match criteria. You should usually proceed from most specific to least specific, but what is most specific ( i.e., most likely to be unique ) may vary depending on your data.

Note: You can test alternative match sequences — you are not stuck with the results of the match process. You can back up and rematch. The match options shown include only those for which you have fields mapped.uploadmatch

If records are unmatched after all passes but possess the matching fields you prioritized for at least one match pass, they will be flagged as possible additions as new constituents. They will be grouped with records containing the same values for the matching fields in the first pass in which they were unmatched. Constituents will not be added unless at least one of the input records so grouped contains a core identifier — last name, first name or email — making the new constituent viable.

Since Version 3.0, WP Issues CRM matches a record to the first valid match it sees. In versions before 3.0, if a record matched multiple records in any pass, it would be bypassed by that and all subsequent match passes.

Read about the next step in the upload process — setting default values.


Specific Fields in the Upload Process

All data for uploads is validated according to the same rules that apply when data is input through a WP Issues CRM save/update form. If a field on a record fails validation, the record will not be uploaded. You will be able to see how this is going to go in the validation dialog after you have mapped fields. The validation process occurs on several levels within WP Issues CRM.

For fields that take a finite set of values (for example, address type), you generally can customize those values under Options, but you should do so before you finalize your upload. If a lot of records fail validation because of an uncustomized option set, you can recustomize the option set and revalidate (as long as you haven’t already proceeded to complete the upload).

ID: WP Issues CRM Internal ID — you will only have this if you started with export from WP Issues CRM.
Street Address: Single field like so: 123 Main St, Apt 1. Do not worry about exactly how you punctuate your addresses or how you refer to street (ST, st, str) or apartment. If you choose under Settings, WP Issues CRM can link to the US Postal Services to standardize records when you view them individually. For batch standardization, you will need an outside service.

In Version 3 and up, you have the option of supplying street address components which, if supplied, will be combined to form street address. The components will be separated by spaces, except that:

  • No space will be inserted between parts 1 and 2 (so they could be used for loading from sources that separate street number and suffix — 35 R will be saved as 35R.
  • Part 7 is intended as a “secondary address identifier” — like “unit” or “apt”. If no part 7 is supplied, but part 8 is supplied (and is non-blank), the abbreviation “Apt” will be inserted before part 8. The address will be unpunctuated (following US postal standards).

Note: You do not need to use the component parts in a consistent away across uploads — different sources split address different ways. You also do not need to use the components consecutively — you might load just components 1, 5 and 8, for a random example.

State: WP Issues CRM does not care how you abbreviate or do not abbreviate state — but it will enforce the definitions you define under Options. Note that you will be able to default state for your entire upload under the Set Defaults tab.
Zip: If Verify USPS Zip Format is set under WP Issues CRM Settings, Postal Code will be validated for 5 or 5-4 character format. Note that you will be able to default zip for your entire upload under the Set Defaults tab. On the constituent form (but not for the upload process), you can configure postal address checking which automatically standardize address format and supply zip code.
City: City is not validated. Note that you will be able to default city for your entire upload under the Set Defaults tab.
Email Address: Email Address, if supplied, must be valid. Validation is through PHP’s FILTER_VALIDATE_EMAIL filter. According to the PHP Manual, “In general, this validates e-mail addresses against the syntax in RFC 822, with the exceptions that comments and whitespace folding are not supported.”
Phone Numbers: Non-numeric characters in phone numbers are dropped on save to the database. When showing a phone number in a form or report, WP Issues CRM will reinsert formatting for 7 or 10 digit phone numbers (if appropriate).
Dates: WP Issues CRM accepts most standard date formats (using the php DateTime object) and date formats can vary within a column. Completely invalid dates will reduce to a blank and be treated as errors.
Gender: You can set any gender coding you wish under Options. You will need to do so before validating your data — codes that have not been set as options will lead to record rejection.
Deceased 0 for living. 1 for deceased.
Address, Email, Phone and Activty types: You will be able to use any existing type codes in your upload by setting Options and/or you can accept default settings for these fields.
Status types: You will be able to use any existing status codes you have by setting Issue/Case Status Options.
Issue: This field must be a valid numeric issue/post ID from your WordPress database (if you specify it). It must point to an issue that is open for activity assignment.
Issue Title: If you specify issue titles that do not exist, you will have the option of automatically creating new issues — see Set Default tab. However, if Issue is supplied, it always overrides Issue Title for matching purposes. When creating issues based on title, the activity assignment status is not checked. Blank titles are not valid and will generate an error.
Issue Content: Issue Content is only used if (a) Issue Title is mapped; (b) Issue Title is not overridden by an Issue setting; (c) You affirmatively elect under Set Defaults tab to add new issues.
Pro/Con You will be able to use any existing Pro/Con or favorability coding that you set up under Options.

Validation Rules in the Upload Process


WP Issues CRM validates input data on multiple levels while giving you considerable flexibility as to the quality and file format of data that you choose to import. WP Issues CRM considers the following questions as it looks at data that you have uploaded and are planning to save in the database.

The upload process can consider each question at two levels:

Does the input file record include enough identifying information to be saved as a new constituent?

Conceptual Rule:

  • You cannot upload a file without mapping at least some constituent fields. If data is missing for some rows for the mapped fields, however, that will not stop the creation of constituents. By contrast, when entering constituents from the form, at least one of name, email or a partial address needs to be supplied.

Column Implementation:

  • You cannot move past the mapping stage without mapping some fields.
  • You cannot run express if you map activity fields. When you do not run express, you have to go through the matching step and that step forces you to have some constituent identifiers mapped for matching.

Item Implementation:

  • No enforcement at the item level. If you inadvertently create incomplete records through with a bad file, you can always back the upload out.

Does the input file include enough information to save valid phone, email, address or activity records for the constituent?

Conceptual Rules:

  • For a phone record, phone number or extension must be present.
  • For an email record, email address must be present.
  • For an address record, some address term — street, city, state or zip must be present.
  • For activity records, issue must be present.

Column Implementation:

  • Enforced at the mapping stage for phone, email and address — not required to furnish phone, email or address, but cannot map phone, email or address type without mapping corresponding data. These rules are enforced in WIC_Entity_Upload_Map.
  • Enforced at the default setting stage for activity. If activity data is supplied, issue must be either mapped from the source file or defaulted. Default values are saved as soon as changed, but upload status is not updated to “defaulted” until all necessary choices are made. This rule is enforced by the function decideWhatToShow() in upload-set-defaults.js.

Item Implementation:

  • The requirement that some data be present on phone, email or address records is enforced at the final upload stage in WIC_Entity_Upload_Complete — phone, email or address subrecords that are missing all values are simply not stored.
  • For activity records, WIC_Entity_Upload_Validate enforces the limited required logic for activities (issue must be present) that is enforced in form submissions but only if issue is mapped. Issue and activity dates can be set later in the upload process at the default setting stage. If post_title is mapped, post_title cannot be blank, but non-blank post_titles that do not already exist on posts can be created automatically at the default setting stage.

Is every submitted data item valid?

Conceptual Rules:

Column Implementation:

  • Not a column level issue — only applies to items for mapped columns.

Item Implementation:

  • Strictly enforced at the Validate Data stage and, in turn, at subsqequent stages: WIC_Entity_Upload_Validate tests each mapped value on each record using the same validation routines that are used in the form context. If any field fails validation, the record is marked as having failed validation.
  • You do have the option to by pass all validation by running express as long as you are not mapping activities.

Is the data from a better source than existing data?

Conceptual Rules:

  • The user can make a decision as to whether input data is at least as reliable as what is already in the database and so whether that data should be allowed to replace what is in the database.
  • If not, then the user can choose to not do updates on the core constituent data (name, date of birth, gender, etc.) and address data. This would make sense, for example, if one were uploading a list of people to whom one wanted to add the same activity. The user might define soft matching ( for example, based on first five characters only) to maximize matching on low quality input data and get the activities loaded, but would want to avoid updating good database records with unreliable name and address data.

Column Implementation:

  • At the default setting stage, the user can check “Protect primary constituent data and address”. If this value is checked, then no updates will be performed for the constituent or address records. However, setting protection on will not prevent the addition of new constituents. This logic is enforced in WIC_Entity_Upload_Complete.

Item Implementation:

  • Setting protection on will not prevent the addition of address information if the address type (set by default, or, if default not set, on the individual record) is not already present on the database. So, if one had reliable home addresses one wanted to protect, one could set protection on, but this would not prevent the upload of rougher work address data mixed in to the input file.

Should blank values overly existing data?

Conceptual Rules:

  • Generally, the answer to this will be no, but it is a user call. One might want to allow blank values to overwrite existing values if one felt that the existing values were unreliable.

Column Implementation:

  • N/A

Item Implementation:

  • At the Set Defaults stage, the user can check “Protect all fields from being overwritten by blank input.” If this value is checked (which is the default), then no updates will be performed to any field that would write a blank into the field for an existing record or subrecord. This logic is enforced field by field in WIC_Entity_Upload_Complete.