gexport issues

General discussion about Plugins for Cacti

Moderators: Moderators, Developers

Post Reply
Author
Message
thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

gexport issues

#1 Post by thrca » Mon Oct 14, 2019 9:56 am

I recently upgraded from cacti 0.8.8g to 1.2.7. Most everything went well, except for the graph exports, which I was heavily using. They were previously built in, but now they were moved into a seperate plugin.
My base url for cacti is myserver.mydomain.com/cacti/ (This is important for later)

I pulled the gexport plugin from github, installed and enabled it. I currently have configured local exports (I was using FTP, but want to make sure the exports are correct first).

I configured as follows:
Method: local
Args: rsync -zav
Theme: classic
Presentation: Tree
Effective User: n/a
Tree: 3 trees selected (I tried "All Trees" which didnt appear to work at all.. 3 trees for testing)
Expand Devices/Sites: On (I was expecting this to create one folder per Tree branch, which it is not.
Default view thumnails: checked
Max Graphs to export: 9999 (setting to 0 as the tip indicates does NOT work)
Export thumbs: checked
Clear dir: not checked
Dir: /var/www/export

The issues I have found so far:
Old export system made a structure where each branch of the tree had a seperate folder, contained within was the graph_x.html for all the devices in that branch, and a graphs folder with the png files for that branch. The new plugin creates all the graph_x.html for all the devices in the root export folder, regardless of the setting of "Expand Devices/Sites" which indicates it should follow the old behavior.

The main.css that is created is referencing URLs for images that do not exist. It has relative URLs to things like "./../../../images/shadow_gray.gif". Clearly, this makes some assumtions about my layout that are incorrect. I can't figure out where they would expect my export to go to match this. /var/www/export/ is the export path, so based on that, the CSS is expecting /images/ to be in /. This won't work. Why is export not creating shadow_gray.gif in the upload process?

Maximum Graphs to Export: Setting this to 0 does not in fact mean unlimited, because when I set to 0, no graphs export and I see "Reached maximum graphs setting of 0" when running poller_export.php manually.


Is there a way to get this working again where I get graphs per branch in my tree? If I need to download a dev version (or older version, etc.) I can.. I also can help with code if there is specific things I should test in code and I can commit them back to the git.


Jim

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#2 Post by thrca » Mon Oct 14, 2019 10:08 am

This appears to fix the issue with graph_max=0

Code: Select all

diff --git a/functions.php b/functions.php
index 03789ac..9a0d3c8 100644
--- a/functions.php
+++ b/functions.php
@@ -776,7 +776,7 @@ function export_graphs(&$export, $export_path) {

                        $exported++;

-                       if ($exported >= $export['graph_max']) {
+                       if ($exported >= $export['graph_max'] && $export['graph_max'] > 0) {
                                db_execute_prepared('UPDATE graph_exports
                                        SET last_error="WARNING: Max number of Graphs ' . $export['graph_max'] . ' reached",
                                        last_errored=NOW()

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#3 Post by thrca » Mon Oct 14, 2019 11:08 am

Addition to be more concise in debug messages about what is going on

Code: Select all

diff --git a/functions.php b/functions.php
index 9a0d3c8..6266bb5 100644
--- a/functions.php
+++ b/functions.php
@@ -690,6 +690,8 @@ function export_graphs(&$export, $export_path) {
                $user = -1;
        }

+    export_debug('Export presentation is ' . $export['export_presentation']);
+
        if ($export['export_presentation'] == 'tree') {
                if ($trees != '0') {
                        $sql_where = 'gt.id IN(' . $trees . ')';
@@ -697,7 +699,7 @@ function export_graphs(&$export, $export_path) {

                $trees = get_allowed_trees(false, false, $sql_where, 'name', '', $total_rows, $user);

-               export_debug('There are ' . sizeof($trees) . ' to export');
+               export_debug('There are ' . sizeof($trees) . ' trees to export');

                if (sizeof($trees)) {
                        foreach($trees as $tree) {
@@ -714,7 +716,7 @@ function export_graphs(&$export, $export_path) {
                                'local_graph_id', 'local_graph_id'
                        );

-                       export_debug('There are ' . sizeof($graphs) . ' to export for all trees.');
+                       export_debug('There are ' . sizeof($graphs) . ' graphs to export for all trees');

                        if (sizeof($graphs)) {
                                foreach($graphs as $local_graph_id) {

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#4 Post by thrca » Mon Oct 14, 2019 11:27 am

Curiously, in function "export_graphs", it was returning 0 for sizeof($graphs)... Further analysis reveals that the tree ids (in my test case) that it was looking for was 24, 63. The table graph_tree_items contains only rows for 22,34. Why the discrepancy?

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#5 Post by thrca » Mon Oct 14, 2019 11:50 am

So the mis-count was because debug was only reporting graphs not associated with hosts.. I made the following adjustments to debugging to account for this.

Code: Select all

diff --git a/functions.php b/functions.php
index 6266bb5..297aa90 100644
--- a/functions.php
+++ b/functions.php
@@ -704,19 +704,18 @@ function export_graphs(&$export, $export_path) {
                if (sizeof($trees)) {
                        foreach($trees as $tree) {
                                $ntree[] = $tree['id'];
+                export_debug('Tree \'' . $tree['name'] . '\' with id \'' . $tree['id'] . '\' is allowed');
                        }
                }

                if (sizeof($ntree)) {
                        $graphs = array_rekey(
-                               db_fetch_assoc('SELECT DISTINCT local_graph_id
+                db_fetch_assoc('SELECT DISTINCT local_graph_id
                                        FROM graph_tree_items
                                        WHERE local_graph_id > 0
                                        AND graph_tree_id IN(' . implode(', ', $ntree) . ')'),
-                               'local_graph_id', 'local_graph_id'
-                       );
-
-                       export_debug('There are ' . sizeof($graphs) . ' graphs to export for all trees');
+                'local_graph_id','local_graph_id'
+            );

                        if (sizeof($graphs)) {
                                foreach($graphs as $local_graph_id) {
@@ -726,11 +725,15 @@ function export_graphs(&$export, $export_path) {
                                }
                        }

+            export_debug('There are ' . sizeof($graphs) . ' graphs not in hosts to export for all trees');
+
                        $hosts = db_fetch_cell_prepared('SELECT GROUP_CONCAT(DISTINCT host_id)
                                FROM graph_tree_items
                                WHERE graph_tree_id IN(?)',
                                array(implode(', ', $ntree)));

+            export_debug('There are ' . sizeof(explode(',',$hosts)) . ' hosts to export for all trees');
+
                        if ($hosts != '') {
                                $sql_where = 'gl.host_id IN(' . $hosts . ')';
                                $graphs = get_allowed_graphs($sql_where, 'gtg.title_cache', '', $total_rows, $user);
@@ -743,6 +746,8 @@ function export_graphs(&$export, $export_path) {
                                        }
                                }
                        }
+
+                       export_debug('There are ' . sizeof($ngraph) . ' total graphs to export for all trees');
                }
        }else{
                if ($sites != '') {

netniV
Cacti Guru User
Posts: 3132
Joined: Sun Aug 27, 2017 12:05 am

Re: gexport issues

#6 Post by netniV » Mon Oct 14, 2019 4:23 pm

If you can submit your change as a PR, we can review and commit that. I'll try and have a look at your other comments tomorrow if I have time.

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#7 Post by thrca » Tue Oct 15, 2019 8:47 am

PR created for the max_graphs issue and some additional logging.. Regarding the old behavior of creating a folder per tree branch, this was a LOT more involved and probably not needed for new users of exports, so I just forked and started re-working this into my version. I depend on this old structure because I have external systems that are depending on it, so I had to reproduce it as close as possible. I may re-write my external systems to use the new structure, if I can figure it out, but requires a bunch of regex parsing of the generated html files.

netniV
Cacti Guru User
Posts: 3132
Joined: Sun Aug 27, 2017 12:05 am

Re: gexport issues

#8 Post by netniV » Tue Oct 15, 2019 6:03 pm

To be fair, I think there may be other users who like the subfolder idea. I don't use it but I know it sounds good to me.

It may be worth having a branch that contains the folder work so it can be compared/pulled if it looks good.

thrca
Posts: 10
Joined: Thu Feb 27, 2014 3:19 am

Re: gexport issues

#9 Post by thrca » Tue Nov 05, 2019 10:21 am

For the most part, I scrapped the branch-per-folder thing and overhauled how my 3rd party system is reading data. The issue I am having currently is related to how the export works. When I have all trees selected (103 trees), it does not successfully export. I have attempted selecting all the trees individually, and somewhere around 63-67 trees, it stops working properly, but before that, it works successfully.

It appears to go through the export correctly, but fails when writing the graph tree data. I have a hunch that it is selecting all the data into an array, but overrunning the amount of memory allocated to each process, so I have to play with the memory settings for PHP to see if I can get it working.

The way I currently have it built for my fork is that I export all the graphs and thumbnails images, then rather than writing all the graph_*.html and stuff, I only am creating a .json file with data about the graphs, which I then ingest using my own system to display the data as needed.

netniV
Cacti Guru User
Posts: 3132
Joined: Sun Aug 27, 2017 12:05 am

Re: gexport issues

#10 Post by netniV » Tue Nov 05, 2019 11:16 am

Sounds interesting. Will check it out when your done to see if I want to pull any of it back.

Post Reply