Implementing Moodle's Privacy API in a Moodle Plugin - Part 4

Continuing the series on implementing Moodle's Privacy API in my questionnaire plugin, I will add code to handle all of the questionnaire data. I did my smaller test set, but now I need to get a fully working implementation.

To start with, I need to add all of the potential response data to my privacy data provider. For questionnaire, this includes multiple question and response tables. For the export, I also need to design an appropriate output structure. This post will work on fulfilling these functions.

I have a couple of options for an output structure. I can create one JSON structure with all of the responses, questions and answers, like this:

{
    "name": "Test Questionnaire",
    "intro": "A wonderful description of the questionnaire.",
    "responses": [
        {
            "complete": "Yes",
            "lastsaved": "Friday, 18 November 2016, 8:14 pm",
            "questions" : [
                {
                    "questionname": "Q1. Car ownership",
                    "questiontext": "Do you own a car?",
                    "answers": [
                        "No"
                    ]
                },
                {
                    "questionname": "Q2. Characters",
                    "questiontext": "Enter no more than 10 characters.",
                    "answers": [
                        "123456"
                    ]
                },
                {
                    "questionname": "Q3. Numbers",
                    "questiontext": "Check all that apply",
                    "answers": [
                        "1,3,5,Another number: 7"
                    ]
                },
                {
                    "questionname": "Q4. Rate course",
                    "questiontext": "Rate these",
                    "answers": [
                        "Formatting your course: Very easy to use",
                        "Laying out your course: Easy to use"
                    ]
                }
            ]
        }
    ]
}

Or I could use subcontexts, and create a directory structure. Something like this:

In this case, each response by the user would have its own directory, with a separate subdirectory for each question and its specific response. This would be done using subcontexts. I feel the first option is really the best choice for what I need. The second one seems like overkill.

Building the directory structure is done in the forum module. I'm not completely sure how it works, but if you walk through the forum's provider file, you can see that it is built through arrays, and exported through nested calls to export_data.

In any case, building a JSON structure like the one I planned above, is not too difficult. I already have a function in questionnaire that returns a structured data set that I use when sending responses via email or other notification methods. It isn't exactly what I need, but is close enough. So I'll modify it and make sure it still works for the code that uses it currently. The modified code is here, and I create another quick function to use in the privacy provider. The rewrite of my export function that provides all appropriate response data now looks like this.

The last thing I require is the full deletion functions. I do have some library functions that delete response data, but they also log events. The core plugins all seem to provide direct database deletions rather than using their deletion library functions. So, I'll do the same, creating one function that does most of the deletion work so that the two privacy API functions don't have to duplicate that code. The end result is here.

I have tested all of this code with the test code Moodle provided. It all seemed to work fine. I'll see if I can create some automated tests into the module's testing code as well, and do more testing before I release this.

If you have any questions about this work, please ask here or in the forums on Moodle.org.

Implementing Moodle's Privacy API in a Moodle Plugin - Part 3

Continuing the series on implementing Moodle's Privacy API in my questionnaire plugin, I will add code to handle deletion of user data.

The documentation indicates that there are two functions to implement. The delete_data_for_all_users_in_context handles deleting all users' data for a provided context when a defined retention period has expired. The retention period is part of the new privacy settings. The delete_data_for_user handles deleting user data for the provided contexts, when a user has requested to be forgotten.

Looking at the examples in the documentation and in the two modules I have been referring to, forum and choice, these functions determine the data records that need to be deleted and then delete them from the database. Doing it this way, instead of using a specific module's API, seems odd to me. I would have thought using the module API would be safer. But it also means that the data is deleted without leaving information about why and how it was deleted. Most API's would log a deleting event in order to have accountability for the activity. It's possible that logging this deletion violates the GDPR's "forget me" policy? I will need to look into this.

For now, I'll follow the same strategy, and create the record deletion code in these functions.

Continuing with my simplified example, using only the attempts table, these functions are very straightforward. The delete_data_for_all_users_in_context function needs to delete all of the questionnaire_attempts records with the questionnaire id of the context passed into the function. So, the code looks like this:

public static function delete_data_for_all_users_in_context(\context $context) {
    global $DB;

    if (!($context instanceof \context_module)) {
        return;
    }

    if ($cm = get_coursemodule_from_id('questionnaire', $context->instanceid)) {
        $DB->delete_records('questionnaire_attempts', ['qid' => $cm->instance]);
    }
}

The delete_data_for_user function needs to delete all data for each provided context for the specified user. The parameter passed in is a new structure, \core_privacy\local\request\approved_contextlist, which contains the user and the context information we need. It provides methods to get the user and context information. Knowing that, the code becomes very similar to the previous function, except that it will delete all of the attempt records with the contexts' questionnaire id's and the specified user id. The code looks like this:

public static function delete_data_for_user(\core_privacy\local\request\approved_contextlist $contextlist) {
    global $DB;

    if (empty($contextlist->count())) {
        return;
    }

    $userid = $contextlist->get_user()->id;
    foreach ($contextlist->get_contexts() as $context) {
        if (!($context instanceof \context_module)) {
            continue;
        }
        if ($cm = get_coursemodule_from_id('questionnaire', $context->instanceid)) {
            $DB->delete_records('questionnaire_attempts', ['qid' => $cm->instance, 'userid' => $userid]);
        }
    }
}

To test these functions, I get the script provided from the Privacy API Utilities. Executing this function allows me to specify a username which will have its data removed. Before I execute this on my test site, I backup a copy of the database. My functions are not complete at the moment and will only delete the "attempts" record, leaving other data intact. If my functions work, I can restore the database afterward.

Executing the test script, shows a lot of output. Searching through that output, I find:

Processing mod_questionnaire (42/515) (Monday, 4 June 2018, 8:44 pm)

which is good. And, when I check the questionnaire_attempts data table, I see that the records for that user have indeed been deleted. Looks like this part of the API is working.

Now that I have the basic version working, I'll go back and make sure I do the complete job.

Looking ahead, I may need to learn about subcontexts, which are used in the forum provider. On the API documentation page, you can see it referred to. I believe its the key concept in creating the directory like structure of an export, as shown in the image below:

Stay tuned for Part 4, where I will determine if this is needed, and figure out how to do it.

Implementing Moodle's Privacy API in a Moodle Plugin - Part 2

In part 1, I began implementing Moodle's Privacy API in my questionnaire plugin, in order to meet the requirements of the GDPR. In this post, I will add the specific code to do this.

I have a skeleton file in place, that includes all of the class and function specifications that I need. Next, I need to describe each data table that includes user data. The questionnaire has several tables that do this, namely:

• questionnaire_attempts
• questionnaire_response
• questionnaire_response_bool
• questionnaire_response_date
• questionnaire_response_other
• questionnaire_response_rank
• questionnaire_response_text
• questionnaire_resp_multiple
• questionnaire_resp_single

It looks like I need to add each of these to the $collection variable. And, each table and relevant field will require a language string, as shown in the documentation example. To start with, I'll implement just the questionnaire_attempts table.

Adding this table to the get_metadata function means defining the relevant fields. In this case, this table stores the user id, the question id, the response id and the time stamp of when the latest submission for this attempt occurred. Each of these fields can be considered private data, although the question id points to the actual question which really only provides context for a specific question response. I'll stay on the side of providing too much information rather than too little and include it. My function now looks like:

public static function get_metadata(collection $collection) : collection {

    // Add all of the relevant tables and fields to the collection.
    $collection->add_database_table('questionnaire_attempts', [
            'userid' => 'privacy:metadata:questionnaire_attempts:userid',
            'rid' => 'privacy:metadata:questionnaire_attempts:rid',
            'qid' => 'privacy:metadata:questionnaire_attempts:qid',
            'timemodified' => 'privacy:metadata:questionnaire_attempts:timemodified',
        ], 'privacy:metadata:questionnaire_attempts');

    return $collection;
}

And, I add each of the privacy strings to the language file as:

$string['privacy:metadata:questionnaire_attempts'] = 'Details about each submission of a questionnaire by a user.';
$string['privacy:metadata:questionnaire_attempts:userid'] = 'The ID of the user for this attempt.';
$string['privacy:metadata:questionnaire_attempts:rid'] = 'The ID of the user\'s response record for this attempt.';
$string['privacy:metadata:questionnaire_attempts:qid'] = 'The ID of the questionnaire record for this attempt.';
$string['privacy:metadata:questionnaire_attempts:timemodified'] = 'The timestamp for the latest submission of this attempt.';

Now that I have added the metadata, I should be able to see them at the "Plugin privacy registry" page of the site. Navigating to that page, and opening the section on questionnaire, I do indeed see the definitions I just added:

Next, I need to provide a way to retrieve and return the list of contexts for which my plugin stores user data. For my plugin, the only context is CONTEXT_MODULE. And I can determine the context module id for each questionnaire a user has responded to by the qid field in the questionnaire_attempts table and joining tables back through the course_modules table to the context table using SQL. My function looks like this:

public static function get_contexts_for_userid(int $userid): \core_privacy\local\request\contextlist {
    $contextlist = new \core_privacy\local\request\contextlist();

    $sql = "SELECT c.id
             FROM {context} c
       INNER JOIN {course_modules} cm ON cm.id = c.instanceid AND c.contextlevel = :contextlevel
       INNER JOIN {modules} m ON m.id = cm.module AND m.name = :modname
       INNER JOIN {questionnaire} q ON q.id = cm.instance
        LEFT JOIN {questionnaire_attempts} qa ON qa.qid = q.id
            WHERE qa.userid = :attemptuserid
    ";

    $params = [
        'modname' => 'questionnaire',
        'contextlevel' => CONTEXT_MODULE,
        'attemptuserid' => $userid,
    ];

    $contextlist->add_from_sql($sql, $params);

    return $contextlist;
}

Next, I need to provide a way to export user data. The documentation doesn't provide an example, but I can find examples in the core code.

There are a number of data types that must be exported mentioned in the documentation, but questionnaire only needs to worry about the "data" part. The documentation section also describes using the \core_privacy\local\request\content_writer but the code examples in the documentation use \core_privacy\local\request\writer. Looking at the /privacy/classes/local/request/content_writer.php file, I can see that is an interface, while the /privacy/classes/local/request/writer.php is a class described as a "factory class used to fetch and work with the content_writer". So I think the "writer" class has been provided as a shortcut.

Looking at the exporter code for choice and forum, it appears that there is no specific format for the output of a module. The data is structured as JSON, but the elements seem to be up to the plugin. This makes sense, since any plugin can have very different data.

For example, a choice activity export looks like this:

{
    "name": "Choice One",
    "intro": "",
    "completion": {
        "state": 0    
    },
    "answer": [
        "Choice 2"    
    ],
    "timemodified": "Wednesday, 3 May 2017, 6:28 pm"
}

While a forum post looks like this:

{
    "subject": "My New Post",
    "created": "Friday, 1 June 2018, 3:15 pm",
    "modified": "Friday, 1 June 2018, 3:15 pm",
    "author_was_you": "Yes",
    "message": "<p>Hi. This is my new post. I hope you like it.</p>
}

Before I implement an exporter, I will need to decide what the data should look like. I'll stick with my simple attempts data for now. Since any questionnaire instance can have multiple attempts by a user, it makes sense to create a structure organized by the instance; in this case the course module id. So my structure should look like this:

{
    "name": "Questionnaire name",
    "intro": "Complete this questionnaire",
    "completion": {
        "state": 0    
    },
    "attempts": [
        {
            "responseid": "rid1",
            "timemodified": "Wednesday, 3 May 2017, 6:28 pm"
        },
        {
            "responseid": "rid2",
            "timemodified": "Thursday, 4 May 2017, 9:31 am"
        }
    ]
}

Looking at the choice activity code for the exporter, I create a function to create the JSON structure I am aiming for. You can see the code here. This code uses several functions provided by the API that are not documented in the wiki. The documentation is really in the class files themselves.

The following line displays the time and date in a readable form:

'timemodified' => \core_privacy\local\request\transform::datetime($attempt->timemodified),

You can find the datetime function in the /privacy/classes/local/request/transform.php file.

The following line gets a structure containing general data for the activity and user that can be merged with the data more specific to the activity:

$contextdata = \core_privacy\local\request\helper::get_context_data($context, $user);

This function is contained in the file /privacy/classes/local/request/helper.php. Following through that code, it creates the part of the JSON structure I need, prior to the 'attempts' array.

The following lines, merge the specific data I want to export with the general data and then writes that JSON data to the export function:

$contextdata = (object)array_merge((array)$contextdata, $attemptdata);
\core_privacy\local\request\writer::with_context($context)->export_data([], $contextdata);

The with_context function is contained in the file /privacy/classes/local/request/writer.php,  and calls the export_data function which is ultimately located in the /privacy/classes/local/request/moodle_content_writer.php file.

The end result of this is an exported structure in JSON form.

Now, to test this, Moodle has provided some scripts that can be created and executed from the CLI. The one I want to use is the "Test of exporting user data" script, provided on that page. So, I create that script on my test site, and execute it. When I execute it, there is a lot of output. Scanning through the output, I see:

"Processing mod_questionnaire (4/15) (Friday, 1 June 2018, 8:37 pm)"

which is positive.

And the last line says:

"== File export was uncompressed to /moodledevsite/moodledata/temp/privacy/3d5750c5-4d5b-4e96-9e86-663cbc9ed177".

This means that there is data located in my moodledata directory, that should contain the exported data. A visual structure of that area looks like this:

I have opened it to the questionnaire I am testing. The "data.json" file will contain the data I exported. When I open the JSON file, I see:

{
    "name": "Test Questionnaire",
    "intro": "<div><p>A wonderful description of the questionnaire.<\/p><\/div>",
    "completion": {
        "state": "1"    
    },
    "attempts": [
        {
            "responseid": "66",
            "timemodified": "Friday, 18 November 2016, 8:14 pm"        
        },
        {
            "responseid": "88",
            "timemodified": "Tuesday, 11 April 2017, 8:50 pm"        
        },
        {
            "responseid": "89",
            "timemodified": "Tuesday, 11 April 2017, 8:54 pm"        
        }
    ]
}

Which appears to match what I wanted.

That's some good progress. In Part 3, I'll add the delete data portion of the API.

Implementing Moodle's Privacy API in a Moodle Plugin - Part 1

The General Data Protection Regulation, or GDPR has now come into effect. Essentially it is a regulation in EU law designed to protect the privacy of online data for individuals within the EU. As such, any online service providers who work within the EU, or have members from the EU, need to address their online data privacy. I won't go into detail about what this regulation is, or how it is interpreted legally, in any articles here, but you can read about it online. Wikipedia has an overview.

For a Moodle plugin developer, this means ensuring your plugin handles user data in accordance with this regulation. A plugin needs to be able to provide the data it stores for an individual user to that user upon request, and it needs to be able to remove a specific user's data if requested. Fortunately, Moodle has provided an API for plugin developers to do all of the heavy lifting for plugins. In this article, I will begin to learn about this API and implement it in a plugin, with the goal of making my questionnaire plugin GDPR compliant.

A great place to start is this video that Moodle HQ put together, featuring the core developer Andrew Nicols explaining how to go about implementing this API. There is also the main Privacy API documentation and the Subject Access Request FAQ.

To begin, I'll create a new branch for this work based on the 3.5 stable branch called M35_PRIVACY_API. My work for this will be tested on a Moodle 3.5 site.

First step is to determine if the plugin actually contains personal data. The privacy API defines what it considers to be personal data in the documentation. I think the following paragraph pretty much indicates that the questionnaire has personal data about each user that completes one:

The most obvious clue to finding personal data entered by the user is the presence of a userid on a database field. Any data on the record (or linked records) pertaining to that user may be deemed personal data for that user, including things like timestamps and record identification numbers. Additionally, any free text field which allows the user to enter information must also be considered to be the personal data of that user.

The questionnaire stores a user's response to all of its questions with a timestamp and the specific answers. This data is definitely personal data.

The documentation indicates that my plugin must implement a relevant metadata and request provider. To do this, I must create a class in the namespace mod_questionnaire\privacy in a file named mod/questionnaire/classes/privacy/provider.php. And, since my plugin does store personal data, the provider class must implement the \core_privacy\local\metadata\provider interface. In order for it to export and delete user data it must also implement a request provider. For an activity plugin, I should implement \core_privacy\local\request\plugin\provider for the request provider.

Note that there are other request providers your plugin might need, depending on whether they use other Moodle systems. The documentation talks about subsystems like ratings and tags, user preferences, and subplugins. Each of these has a different request provider interface that should be implemented. David Mudrack pointed out a document that listed these interfaces which helps.

 To be complete, my new class must implement the get_metadata, get_contexts_for_userid, export_user_data, delete_data_for_all_users_in_context, and the delete_data_for_user functions.

So, that becomes my first step. I create the mod/questionnaire/classes/privacy/provider.php file as follows:

<?php
namespace mod_questionnaire\privacy;

defined('MOODLE_INTERNAL') || die();

class provider implements
    \core_privacy\local\metadata\provider,
    \core_privacy\local\request\plugin\provider {

    public static function get_metadata(\core_privacy\local\metadata\collection $collection):
        \core_privacy\local\metadata\collection {
        return $collection;
    }

    public static function get_contexts_for_userid(int $userid): \core_privacy\local\request\contextlist {
        $contextlist = new \core_privacy\local\request\contextlist();
        return $contextlist;
    }

    public static function export_user_data(\core_privacy\local\request\approved_contextlist $contextlist) {}

    public static function delete_data_for_all_users_in_context(\context $context) {}

    public static function delete_data_for_user(\core_privacy\local\request\approved_contextlist $contextlist) {}
}

This gives me the basic skeleton to work from.

At this point, I can verify if the plugin API is seen by the Moodle site. To do this, I copy the new work into my development site's plugin directory. I can then go to the "Plugin privacy registry" page of the site, to see if my plugin shows up. This page is in the "Site administration / Users / Privacy and policies" section. On this page, I can open the "Activity module" section and scroll down until I see my questionnaire plugin. If it doesn't have a non-compliant icon next to it, then I have succeeded in making the API visible to Moodle. When I look, I see that I am moving in the right direction. The image below shows my plugin, and another plugin that does not have the API defined yet (on my site anyway).

In part 2, I'll begin adding the code to complete this work.

Looking at Moodle's new plugin development API's - Part 1

David Mudrack, of Moodle HQ released a plugin called the "My Todo List" block. While functional, its real purpose is to show off the latest advanced coding techniques available in recent versions of Moodle. See his Tweet. From the readme:

The main purpose of this plugin is to demonstrate usage of advanced coding techniques available in recent Moodle versions. Most notably:

• Rendering HTML output via Mustache templates.
• AJAX based workflow of the elementary CRUD operations.
• Organising JS into AMD modules.
• Organising external functions into traits.
• Low-level access to the database via persistent models.
• Using exporters for handling the data structures in rendering and AJAX.

For this post, I will install and examine the code with the goal of learning better how to use these concepts.

While I have been a plugin developer for many years, I have not been able to keep up with all of the latest techniques to enhance the plugins I maintain. I am personally really interested in getting better with AJAX and AMD, as many of my plugins could benefit from these features. I have already experimented and released code with renderers and Mustache templates (see the series beginning here), so I won't go deep into those unless I see something new.

To begin, I fork David's repository into my own Github repository. This gives me place to install and play with the code. Once I have a local copy, I load it into my IDE (I use PhpStorm) and take a look at the code.

A quick perusal of the code shows that this is indeed a simple plugin, and uses the techniques I'm most interested in! The main block code file,  block_todo.php, contains five methods, four of which I'm familiar with from standard block development: init(), get_content(), specialization(), and applicable_formats(). But the fifth, get_required_javascript(), I am not familiar with and it is not defined on the block development page. I suspect this has to do with the AMD module feature. And, while that may be true, it appears that it calls the parent method, meaning this function is part of the block class. I need to do some searching to see why this function is there.

I search the development documentation on Moodledocs, and find this function referred to on the "jQuery pre2.9" page. So, it must be a function that has remained even with the new AMD module addition. I think I may want to look for the AMD Module documentation first.

I find a page in the Moodle development docs wiki on Javascript Modules. This appears to be the main documentation for using AMD in plugins. Skimming through I see that initializing an AMD javascript function is done with the $this->page->requires->js_call_amd() function. This is done in the ToDo block's get_required_javascript method.

A closer look at the parent block class, defines the get_required_javascript method as:

Allows the block to load any JS it requires into the page.
By default this function simply permits the user to dock the block if it is dockable.

So, I'm not sure that this function is actually required, but may just be a convenient place to call the js_call_amd function from. I search all of the core blocks for js_call_amd and find two, both of which call it from the get_required_javascript method. So this seems to be the place to do that. Maybe later, I'll play with that to see if that is a required way to do it.

In any case, the function passes the arguments:

'block_todo/control', 'init', ['instanceid' => $this->instance->id]

This coincides with the file amd/src/control.js, which is the expected location within a plugin to find the AMD javascript files.

Getting back to the function of the plugin, the main block get_content function, is a very simple function that does two things: gets a list of todo items, and outputs them on the screen. The technique uses one technique I'm familiar with, templates, and two I am unfamiliar with, Moodle persistents and Moodle exporters.

This is what I am seeing in get_content:

// Load the list of persistent todo item models from the database.
$items = block_todo\item::get_my_todo_items();

This code uses the persistent class, which appears to be a new technique to manage database data using CRUD techniques. It will be interesting to learn why this exists and when it should be used.

// Prepare the exporter of the todo items list.
$list = new block_todo\external\list_exporter([
    'instanceid' => $this->instance->id
,
], [
    'items' => $items,
    'context' => $this->context
,]);

This code uses exporter class and appears to be a technique to manage data passed in and out of web service functions, used by external applications and AJAX.

// Render the list using a template and exported data.
$this->content->text = $OUTPUT->render_from_template('block_todo/content',
    $list->export($OUTPUT));

This code is using a Mustache template to display the block content and utilizes the exporter class to send the data rather than renderers defined for the plugin, as I am used to. Again, I will need to discover why this technique is used.

I have some concepts to learn, so I install the block just to see what it does and how it works. After adding it to my dashboard page, I play with it, and see that it is as advertises, a simple "To Do" list.

Hovering over the controls shows no obvious links, and a quick perusal of the page code shows a form, but not a standard functioning Moodle form. It looks like all of the controls are using Javascript, AJAX and web services to do the actual work. This will be all new to me. :-)

Next post, I will start my functional learning.

Things I learned upgrading to Moodle 3.3

As the Moodle 3.3 launch approached, I decided to check on a few of my plugins and ensure they were ready for 3.3.

The good news is that having had everything pretty much up to 3.2 standards, the effort to reach 3.3 was minimal. In fact, with the two plugins I worked on for the coveted "Early bird 3.3" award, questionnaire module and oembed filter, I could have submitted them as is and they would have passed fine. But, that's not my way, and besides, it would have made this post even shorter than it is.

For the questionnaire module, I first installed the 3.2 version on a new Moodle 3.3 test site, and ran the unit tests and Behat tests. Having automated testing is a great way to manage upgrading, as these tests will hopefully uncover any problems introduced by the Moodle upgrade or by any changes made to the plugin for the new version. In the questionnaire case, I had made a number of improvements to the code, continuing my efforts of paying down "technical debt" and making the code easier to maintain going forward.

The unit tests ran fine, but the Behat tests failed. Additionally, there were messages being kicked out by the developer level debugging.

There were two issues in Behat  I needed to fix. The first is a change to the Behat tests in Moodle 3.3. Where previously a step would say:

'And I follow "Course 1"' 

For 3.3, they need to say:

'And I am on "Course 1" course homepage'

This is due to the new course dashboard introduced in 3.3 that replaces some of the navigation in 3.2.

The other issue I needed to change was places that would say:

'I navigate to "Questions" node in "Questionnaire administration"'

Now need to say:

'I navigate to "Questions" in current page administration'

This was actually a change in 3.2 that didn't affect me until 3.3. I believe I was supposed to have made the change in 3.2, but now it is mandatory.

Making those changes fixed the Behat errors.

The debugging warning I was getting was:

pix_url is deprecated. Use image_url for images and pix_icon for icons.
    line 267 of /lib/outputrenderers.php: call to debugging()
    line 182 of /mod/questionnaire/classes/questions_form.php: call to renderer_base->pix_url()
    line 204 of /lib/formslib.php: call to mod_questionnaire_questions_form->definition()
    line 32 of /mod/questionnaire/classes/questions_form.php: call to moodleform->__construct()
    line 153 of /mod/questionnaire/questions.php: call to mod_questionnaire_questions_form->__construct()

This is telling me that places I am using the function pix_url will eventually (in some future release) fail. And they should be replaced with either pix_icon or image_url. For my case, image_url is the easiest and correct fix. The previous function, pix_url, returned an URL to be used in the output. The new function, image_url, likewise returns an URL. The new function pix_icon returns the entire line of code including the image tags. Using it would require a significant amount or re-coding, and in reality, I am not using icons.

Once I have completed those changes, everything seems fine with questionnaire.

For the Oembed filter, I ran into Behat issues caused by the HTML that is output with 3.3. There has been changes made to some of the core editing features, that changed the xpath layout for the Oembed editing page. As such, I was getting an error like:

001 Scenario: Admin user carries out various provider management tasks. # /Users/mikechurchward/www/moodlehq.git/filter/oembed/tests/behat/management.feature:26
      And the provider "Vimeo" is disabled                              # /Users/mikechurchward/www/moodlehq.git/filter/oembed/tests/behat/management.feature:36
        The "//td/a[text()='Vimeo']/parent::td/div/a[contains(@class,'filter-oembed-visibility')]/i[@title='Show']" element does not exist and should exist (Behat\Mink\Exception\ExpectationException)

The Behat tests for the filter had a custom test called "the_provider_is_disabled" which depended on a specific xpath output that was now no longer happening. This required me to rewrite the Behat function to change the xpath definition to reflect the new output. Manual testing of the function proved that the actual functionality had not been broken, and once the new xpath definition was in place, the Behat tests passed as well.

And that was it. Both plugins were ready for 3.3 and made it in time to get the early bird award. As I continue with other plugins, I will document any new changes required by 3.3 I find in new posts.

Add mobile support to your Moodle plugin - part four

Part Four - Adding More Services

In the previous part of this series, I modified my mobile addon so that clicking on a plugin instance loaded and displayed a "Hello World!" screen. Now, I will add some services so that it will retrieve the number of responses made to the particular questionnaire and display them, when a user clicks on an instance. I'm also going to add a link handler, that will handle the questionnaire when it is selected by a link in other site content.

First, I'm going to add the link function. In all of the other addons I've looked at, a handler was defined for links. In the moodle.org forum, I asked about this, and learned that it was so the mobile plugin was handled properly if it was reached from a link other than the course instance.

Adding the link function is apparently very easy now. The latest release of the mobile app has a simplified handler function that manages the links for me. I really just have to add one line to my main.js file and one line to my services/handlers.js file, as follows:

main.js:

.config(function($mmCourseDelegateProvider, $mmContentLinksDelegateProvider) {    $mmCourseDelegateProvider.registerContentHandler( 'mmaModQuestionnaire', 'questionnaire', '$mmaModQuestionnaireHandlers.courseContent');    $mmContentLinksDelegateProvider.registerLinkHandler( 'mmaModQuestionnaire', '$mmaModQuestionnaireHandlers.linksHandler');

services/handlers.js:

self.linksHandler = $mmContentLinksHelper.createModuleIndexLinkHandler( 'mmaModQuestionnaire', 'questionnaire', $mmaModQuestionnaire);

Now, when I create a link in my course that goes to a questionnaire instance, instead of launching in the browser, it uses the questionnaire mobile plugin.

In the last iteration, I left the isPluginEnabled function in a state where it always returned true. Now, I need to make it do something that actually checks with the site to see if the plugin really is enabled. This will require changing this function in the mobile plugin, and adding a web service to the questionnaire module.

Both the certificate and feedback call the 'mod_[pluginname]_get_[pluginname]_by_courses' web service, so I'll start there. Note that feedback also calls mod_feedback_get_feedback service, but I'll look at that later.

Starting with the mobile plugin, I'll update the services/questionnaire.js::isPluginEnabled function as follows:

self.isPluginEnabled = function(siteId) {
    siteId = siteId || $mmSite.getId(); 

    return $mmSitesManager.getSite(siteId).then(function(site) {
        return site.wsAvailable( 'mod_questionnaire_get_questionnaires_by_courses');
    });
};

What this code does, is verify that the questionnaire module is enabled on the Moodle site by checking for the availability of the web service, mod_quesrionnaire_get_questionnaire_by_courses, on the Moodle site. This means I will need to create that web service in the main questionnaire activity plugin. For this one, I am going to copy from the certificate module, but simplify it down for now to only provide the basic module information and only check for 'view' capabilities.

To add the web service to my module, I do the following:

Add the necessary external services to the classes/external.php file:

This requires creating the file with the external class extending the core external_api class in the mod_questionnaire namespace. In that class, I need to add the function for get_questionnaires_by_courses as well as two other helper functions, get_questionnaires_by_courses_parameters and get_questionnaires_by_courses_returns. These three functions are required to fully define the web service as described in the web services developer documentation.

It is the job of this function to return result information for all questionnaire instances in the identified courses. You can see the full code for this here.

Add the db/services.php file to define the services available by web services:

Web services need to be described in this file. When a plugin is installed or upgraded, the Moodle system uses this file to add the services described in this file to its known, available services. This is more fully described here. Once the Moodle site knows about them, it can allow them to be used by external systems.

Since I already have the questionnaire module installed on my Moodle site, I will need to bump up the version number in my version.php file, to trigger a module upgrade, and get the Moodle site to load my new web service. Once I have done this, I should be able to see that my new service has been installed by going to my site's admin/webservice/service_functions.php screen and looking for the web service there. I upgrade my site with new version and check for the service on the admin screen. Successfully, I see the following:

Since I'm writing web services, I might as well create one that provides some meaningful information from the Moodle site that I can display in my mobile plugin.

Currently, when I click on a questionnaire in the mobile app, I just get the "Hello World!" screen. I am going to modify it so that it tells me how many responses I have made to that instance. To do that, I will need a web service in the Moodle plugin to provide that information back to the mobile plugin.

Back at my db/services.php file, I define a new service called 'mod_questionnaire_get_user_responses' and define it as a service that returns 'a count of the current user responses'.

Then, I code the new service in my classes/external.php file. Its a simple function, that simply returns the count of responses for the user / questionnaire instance in question.

Lastly, I perform a version bump to force an upgrade and verify that the service has been added to the Moodle site:

Now, I need to modify the mobile plugin to use this new service and display the information appropriately. Starting with templates/index.html file, I modify so that it will display a count of the user's responses:

<ion-view>
    <ion-nav-title>{{ title }}</ion-nav-title>
    <ion-content padding="true" mm-state-class>
        <mm-course-mod-description description="description"></mm-course-mod-description>
        {{mymessage}}<br />
        You have {{responses}} responses.
    </ion-content>
</ion-view>

The template will look for a value in the {{responses}} tag, which means I need to add code to the controllers/index.js file to populate that. The code includes a function that uses the promise mechanism to add the response count to the template scope variable, and a helper function to call the plugin service function, $mmaModQuestionnaire.getUserResponses located in services/questionnaire.js.

In the services/questionnaire.js file, I add the code that will call the new web service I created in the Moodle plugin. This is the part that will actually call the site service and extract the response information. The actual call to the web service is in this line:

return $mmSite.read('mod_questionnaire_get_user_responses', params, preSets).then(function(response) {

This should be all I need to change the behaviour of the questionnaire module in the mobile app, so that it will also show me the number of responses.

With my new code added to the mobile plugin, and the new services added to the Moodle module, I need to verify that I can still access the questionnaire in the mobile app, and that I can see the number of responses.

I ramp up my mobile app, click the questionnaire instance, and achieve success:

I still have a lot of work ahead of me to make my questionnaire plugin functional in the mobile app, but I at least now have a good working understanding of what needs to be done, how to use web services, and how to make them useful in the mobile plugin. I will get back to this (eventually), and document my learnings in future posts.

Add mobile support to your Moodle plugin - part three

Part Three - Starting to Build the Moodle Mobile Add-on

In the previous part of this series, I created just enough code to see an impact on the mobile app. Now, I want to begin to actually do things that will lead me to a solution.

To start with, I decided to take an introductory lesson in AngularJS. If you are not familiar, I highly recommend Code School's Shaping up with AngularJS. This helped me to understand a lot of the javascript I am looking at with other mobile add-ons.

Learning this little bit, has helped me to understand the code I am looking at, and helps me to make good guesses about what is going on with the Moodle specific code.

I'm going to use some existing plugins as a starting point for the code I need. Specifically, the certificate module, which is a third party activity plugin, and the feedback module, which will be included in core.

The main.js file defines the necessary parts to the mobile app. The stateProvider configuration sets up the module for displaying when it is clicked in the course. In the certificate code, you can see a controller and a template defined to do this:

controller: 'mmaModCertificateIndexCtrl',templateUrl: 'addons/mod/certificate/templates/index.html'

In AngularJS, this indicates that the display setup code will be in controllers/index.js and that will use templates/index.html to display that code. When a user clicks on a questionnaire instance, these will be the main pieces to determine what is displayed.

The other part of the main.js file, registers the various handlers needed by the mobile app. In the certificate example, you can see there are two: a "course delegate provider" and a "links delegate provider". The course delegate provider is the part responsible to provide functionality for the module to that mobile app. Its job is to handle a click, and use the module plugin code. Without this, the mobile app will display the "not available" message. The links delegate provider provides function for other links to a module instance, not from the main course display. For example, if a link is put into a forum post. Without this, a link will simply launch a web link in the default browser.

For my module, I'm going to focus on the course delegate provider first. And I will use the index files to affect what is displayed in the mobile app.

For this part of the discussion, I have posted my files here. This is the minimum I have determined I need to have my plugin work in the mobile app without any errors.

To start with, I have created my main.js file to define that the questionnaire plugin should be available, that it uses an index controller and that it has a course content handler. I have taken what I started in the last post, and extended out to provide what is absolutely required.

Next, I will further flesh out my services/handlers.js file. In the first iteration, I only provided an isEnabled function, and it simply returned true. Now I have improved that to call a specific function provided by my questionnaire handler:

self.isEnabled = function() {
    return $mmaModQuestionnaire.isPluginEnabled();
};

That function will eventually call web services from the Moodle site to verify that the module is enabled.

I have also added a getController function, which is required by the framework to provide information for displaying and launching my plugin code, as follows:

self.getController = function(module, courseid) {
    return function($scope) {
        $scope.title = module.name;
        $scope.icon = 'addons/mod/questionnaire/icon.svg'
        $scope.class = 'mma-mod_questionnaire-handler';
        $scope.action = function() {
            $state.go('site.mod_questionnaire', {module: module, courseid: courseid});
        };
    };
};

This code tells the framework that my main handler class will be located in services/questionnaire.js and will be called mmaModQuestionnaire. I have also added an icon to be displayed on the course page, and indicated that it will be located in the root of my plugin code. I copied that icon from the main questionnaire plugin. When I run the mobile app with this code, I should now see the questionnaire icon, rather than the default icon, on the course page.

The changes I have made here, mean that I need to create a services/questionnaire.js file, with an mmaModQuestionnaire class, that provides an isPluginEnabled method. You can see that file here. For now, I am keeping the function simple:

self.isPluginEnabled = function(siteId) {
    siteId = siteId || $mmSite.getId();
    return Promise.resolve(true);
};

The function returns a Javascript Promise (if, like me, you aren't familiar with that, this helped me). Essentially, somewhere in the framework, there is code that will call isEnabled in my course handler and will expect a Promise construct. Since I have not written the necessary web services on the other end to verify that, I am simply going to return a resolved promise to indicate my questionnaire plugin should work.

The last parts I need to create are the index display handlers I described in the main.js file. I have created the controllers/index.js file here, and the templates/index.html file here.
As I mentioned previously, these files work together to define and display what is output when a user clicks on a questionnaire instance in the mobile app. For now, I will set it up to simple display "Hello world!".

The key elements in index.js are the properties of the $scope variable:

$scope.title = module.name;
$scope.description = module.description;
$scope.courseid = courseId;
$scope.mymessage = "Hello World!";

These are the bits that I can use as variables in the index.html file to display specific information. In Angular, this means parts contained in "{{}}" or defined as tag elements. The template I create looks like this:

<ion-view>
    <ion-nav-title>{{ title }}</ion-nav-title>
    <ion-content padding="true" mm-state-class>
        <mm-course-mod-description description="description"></mm-course-mod-description>
        {{mymessage}}
    </ion-content>
</ion-view>

The {{ title }}, <mm-course-mod-description description="description"> and the {{mymessage}} elements should be replaced by the corresponding $scope element.

All of this should be enough for me to reload my mobile app questionnaire plugin code into the mobile app, run it and when I click on a questionnaire link, see a page that says "Hello world!". So I copy my code into the app www/addons/mod/questionnaire directory, run the server and check out my app.

The course page now displays this:

Note, that my questionnaire icon is now displaying, rather than the default puzzle piece. Clicking on the questionnaire instance, shows me this:

Which shows me the title and my "Hello World!" message. Looks like I have succeeded.

I'm going to leave it there for this post. In my next post, I will build some services to interact with, and build out the link delegate handler as well.