Improving the delivery of the Page |
Improving the delivery of the Page
Evidence
|
Recommendation
|
---|
- Using a full year's worth of data, and investigating for changing patterns over time, some post characteristics were clearly identified as factors which would likely increase reach and/or engagement.
|
Posts should be made earlier in the workday, and be interactive and/or humourous. Additionally, more posts should be conversational with public health messaging.
|
- Analyses identified that 4 to 7 posts per day were ideal to optimize reach, while 4 or more posts were ideal to optimize engagement.
- For each additional post, up to six, staff spent more time planning per day.
|
Staff should aim to typically post 4 to 5 times per day.
|
- Posts with embedded links have the highest reach.
- Posts with embedded photos possibly have the highest engagement.
- By the end of the year, status updates posts may have been the best type of post to optimize reach and engagement.
- Facebook regularly changes the algorithm that determines which posts are displayed on users' newsfeeds, but these algorithms are typically dependent on other users reading or engaging with the post.
|
Until more is known regarding the objectives of the Page (i.e., if reach or engagement is preferable), staff should aim to have a balance of all three posts types to optimize both reach and engagement.
|
- Three out of the 18 OPHS themes are discussed every month. These three are broad, which is likely why they are discussed regularly, but there are still many themes that are not covered as often as they could be.
- The frequency of discussion of most themes is inconsistent over time.
- The type of audience who engages with specific themes is different.
|
Many themes could be covered more frequently. To aid staff in covering all themes across time, a method to track their usage in real-time should be built.
|
|
Optimizing the maintenance and evaluation of the program |
Optimizing the maintenance and evaluation of the program
Evidence
|
Recommendation
|
---|
- Research staff had to retrospectively code posts of the day into OPHS themes.
- When tracking staff resources over the course of the year, data quality issues (i.e., missing records) were prevalent.
|
For efficiency and accountability in future evaluations the Access charting database needs to be updated to have mandatory tracking of key pieces of information.
|
- Data quality issues within the Access database might be related to other staff covering for the lead PHN.
- Continuous monitoring provides recommendations for best practices in maintaining the Page.
- Previous recommendations have included changes to the Access database to capture more information and to improve quality.
|
Ongoing training and support is required for Reproductive & Child Health team staff to ensure they are up-to-date on best practices for posting and charting. How Page operation protocols and policies can be used most effectively to guide staff work should also be considered.
|
- Many KFL&A Staff actively engage with the CBFB Page from their personal accounts. They may also be the target audience of the Page.
- The active engagement by these staff should aid in promoting the Page to their Facebook friends, but staff behaviours are fundamentally different than external users.
|
When conducting continuous monitoring, KFL&A Staff must be considered separately as their interactions disguise the true level of engagement with the Page.
|
- Fans account for a majority of the reach and engagement.
- A significant majority of fans were recruited on days with paid advertising.
- Over the year, during periods of advertising, the daily fan growth had been decreasing. This may be an indication of saturation. If this is true, future Facebook advertising may be less cost effective.
- As the number of fans increases so does the reach, and to a smaller extent, the engagement.
- Residents of Kingston and females aged 24-34 within KFL&A are over-represented in the fans of the Page.
- Paid advertising was discontinued for four months when the Communications Specialist Position was vacant (who was the only person with the authority to facilitate paid advertising).
- After the unplanned, extended period without advertising, many additional monitoring analyses were possible, which allowed key questions to be answered.
|
Paid advertising through Facebook should be utilized to increase fans, and targeted advertising to under-represented populations should be considered. To ensure advertising can be responsive to the needs of the Page, staff other than the Communication Specialist(s) should be able to control advertising However, Facebook advertising should be continued in planned cycles to allow for continued monitoring of its' effectiveness. It may also be beneficial to investigate advertising the Page through other forms of media.
|
|
Next steps for mainting and evaluating the Page |
Next steps for mainting and evaluating the Page
Evidence
|
Recommendation
|
---|
- Quarterly reports shared in August, December and March provided recommendations based on initial evidence.
- Many of these recommendations were implemented as could be seen in investigations over time; these may have resulted in more effective service delivery.
- When conducting quarterly reports there was delay for analysis and sharing of results, which limited the time that program staff had to implement changes before the next report.
- As the pilot year progressed, some trends were seen to change, including which characteristics resulted in an increase or decrease in reach and/or engagement.
- Even with the development of semi-automated analyses, conducting evaluations/continuous monitoring of the Page is research staff-intensive.
- There was typically insufficient data quarterly to conduct thorough analyses.
|
Continuous monitoring plays an important role in informing service delivery and should be conducted regularly to capture changes in how the Page operates and/or the behaviours of its' users. However, it is not optimal to report quarterly on the Page. Instead, monitoring should be conducted at 6-month intervals during the initial launch of Facebook pages, and every year thereafter.
|
- Evaluations conducted as part of continuous monitoring have been able to answer some key questions about who uses the Page and how they use it, and identify factors which likely increase reach and engagement.
- There are a number of questions that would be answerable through employing 'experimental' conditions during the operation of the Page, such as:
- Do different messaging styles have an impact?
- Are non-public health promotion posts creating 'a Facebook community' that also interacts with promotion posts?
|
In the future, research and program staff should consider applying controlled 'experimental' conditions to test previously identified factors and to investigate as-of-yet unanswerable questions.
|
- In this study, a large number of metrics were used that investigated a whole range of inputs and outcomes.
- The large number of metrics is directly related to the quandary of big data - a mass of information is available for analysis but we cannot control what data is available. Hence, we are unable to ensure the data collected will answer the questions we need answered.
- None of the data automatically collected can directly answer how reach and/or engagement with the CBFB Page translates to a change in users' knowledge and/or behaviours.
|
An outcome evaluation focussed on the users of the Page (their perceptions, attitudes and behaviours) is required. The results from the current evaluation will inform the outcome evaluation.
|
- Much of the audience of the CBFB Page will only view one post in any given time period. Many users will likely only engage with only one post in a lifetime. Currently, based on the number of posts that contain promotion messages, there are fairly equal odds that this post will or will not contain public health promotion messaging.
- The audience of the Page exhibits a wide range of behaviours, with a small subset of fans who view or engage with the Page with relatively higher frequency. However, all users engage with the Page unpredictably.
- Users of social media sites respond to a variety of visuals (photos, videos, etc). Currently, staff on the Page have to use photos from other sources, and have little capacity to embed videos in posts.
|
The content and quality of each post should be considered by staff maintaining the Page. Efforts should be made to improve levels of engagement. In order to increase the appeal of posts, the full range of visual media should be used. This requires support from the Agency in the form of graphic designers and approval to produce low quality videos on topics relevant to the Page.
|
- As a pilot program using a novel medium, with minimal evidence the initial objectives of the Page were very broad.
- While not designed as either a developmental or outcome evaluation, the results from this evaluation do provide context as to what may be reasonable for the Page to achieve. Notably, the frequency in how often users see posts from the Page informs reasonable timelines for changing user's health perceptions, attitudes and knowledge.
|
The program objectives (and correspondingly the logic model) should be slightly adapted based on current evidence.
|
- As of October 2 2015, the CBFB Page pilot has required 403 staff hours to plan, 538 staff hours to maintain, and 1059 staff hours to evaluate. The total investment into this pilot program is estimated to be $103,374.95.
- By the end of the pilot year, after $1664 spent on paid advertising (over nine months), there were 809 fans of the Page.
- In the last quarter, the Page averaged approximately 400 regular viewers.
- In the final month, there were 85 users who actively engaged with the Page, of these, 31 did so more than once.
- There are no available benchmarks to compare the reach and engagement of this Page to other similar pages.
- Assuming that current trends in staff time stay the same, it is predicted that to continue to maintain the CBFB Page would require 410 staff hours a year (1 day per week). To conduct yearly evaluations would require 230 staff hours.
|
A cost-benefit decision must be made to determine what size of audience is required to justify continued maintenance and continuous monitoring costs of the Page, accounting for already invested costs. This decision should be informed by an outcome evaluation that can investigate the link between being reached and/or engaged with the Page, and health knowledge and/or behavioural change.
|
|
Agency Wide Implications |
Agency Wide Implications
Evidence
|
Recommendation
|
---|
The data quality issues in tracking staff time likely resulted in inaccuracies in the total amount of staff resources dedicated to the project.
There was no available data on time spent by some types of staff (i.e., communication staff, the manager of Continuous Quality Improvement, members of Executive Committee).
There was no formal method for tracking staff time outside of the charting database, (except for research staff after November 2014).
The hours and total cost of the pilot year of the CBFB Page cannot be compared to costs of similar programs, because economic analyses have not historically been conducted.
|
In order to appropriately assess the cost-benefit of Facebook as a knowledge exchange medium, economic analyses should be considered in future program evaluations to allow for comparisons between programs within the Agency. To accomplish this efficiently, a formalized system that can be used Agency-wide is required to track hours dedicated to specific projects (similar to the one used currently by the Knowledge Management Division).
|
The CBFB Page is maintained by the Reproductive & Child Health Team whose topic areas cover, many, but not all health topics relevant to the target population. For example, the content experts on communicable diseases and immunizations are located in different teams in a different division within the Agency.
Within the Reproductive & Child Health Team, one staff member is designated as the lead for the Page. This staff member has developed expertise in social media communications, but is not the content expert for all topics within the team's portfolio.
|
Formal policies and procedures and informal supports are required across the team, division, and agency to support the sharing of all content relevant to the target population of the Page. Given the expertise required to craft social media messages, the current structure of the program is appropriate, but engagement from all relevant staff is required to provide post content. Additionally, understanding of the different approval processes required for social media is necessary across management levels.
|
Despite the target of the Page being residents of the KFL&A region, almost 20% of fans identified their location as outside this area; notably, 5% identified their location as within neighbouring LPHAs.
Many locations in neighbouring LPHAs are commonly reached by the Page posts. For many posts with higher reach, that reach is from areas outside of KFL&A.
All of the health promotion messages are appropriate for wider audiences, and many of the more local community links are also accessed by residents of nearby LPHAs.
|
To optimize resource use within the Agency and across agencies, an expansion of the Page and/or the sharing of resources with other LPHAs to serve a wider geographic community should be considered by program staff and managers. Additionally, collaboration could allow for more in-depth experimental testing, and hence more evidence-based operation of the Pages.
|
The planning process for this Facebook page took 403 hours of staff time over three years. During this time, the Agency was reviewing their communications strategy and its risks.
Significant investment has been made in the protocols and evaluation procedures for a health promotion and knowledge exchange Facebook page.
Facebook, and social media more broadly, is a rapidly changing environment. For example, over the course of the protocol development and pilot year, Facebook made significant changes in data availability and methods for accessing them.
The usage of Facebook changed rapidly over the three years the Page spent in development, and in the future the changing demographics of Facebook may make it a non-ideal medium for health promotion and knowledge exchange, while other social media forms may become more attractive.
|
There is considerable risk in the investment in social media for health promotion. A key mitigation strategy is building a high level of adaptability into the operation and evaluation methods. This will likely require systematic changes in policies and procedures within the Agency. For example, mechanisms need to be in place to facilitate the sharing of training and lessons learned on using social media across the Agency to capitalize on previous investment and decrease the planning time needed for future programs. Stream-lined and modular approvals processes, alongside the ability to set priorities for time-sensitive changes, are required to allow staff to be responsive to changes in the social media environment. Finally, in order to optimize the use of resources required to develop and maintain social media presence in the changing environment, multiple platforms should be considered.
|
|