Monday, August 14, 2023

Hackathon Posts - Oracle APEX - Low Code App Platform

Few posts planned for use in expected Hackathons.

Oracle APEX (formerly known as Oracle Application Express) is a very popular Low Code App Platform by Oracle. It is used for developing and deploying cloud, mobile, and desktop applications. The platform offers a web-based integrated development environment (IDE) with a range of features including wizards, drag-and-drop layout, and property editors to simplify the process of building applications and pages.

APEX is a fully-supported no-cost feature of the Oracle Database and can be installed anywhere Oracle Database runs. APEX is also offered on Oracle's Cloud across various services including Autonomous Database Cloud Services and the stand-alone fully managed APEX Application Development service.

 

How to start?

If you have an existing Autonomous Database, you can access the Database Actions - View all database actions and select APEX.




If you don't, you can create an Autonomous database for APEX workload type.


How to learn?

There are lots of great tutorials in Oracle LiveLabs (the site itself is created with APEX):

For starter consider one of the following (first 2 available also on LiveLabs sandbox):

 

For Specific issues there are lots of blogs for the topics and also the following tutorials on LiveLabs: 


There several specific cases examples such as 
 

  

You can also check this Videos page here there many specific tips there.

 

This is the documentation page - https://docs.oracle.com/en/database/oracle/apex/index.html

 

For local installations this is the installation guide.


Friday, August 11, 2023

Oracle Spatial - loading geographic data - Line

 Following the previous post - Oracle Spatial - SQL based Geographic data - Intro, few examples of loading lines of geographical data. Line is usually used when we have routes.

I will cover here:

  • Manual creation
  • Loading GeoJSON

Simple example - Manual Creation of line based on few points

Step 1 & Step 2: Lets assume we have 3 geographic data points that creates a line. 

I will create a table with few descriptions and the line:

create table Line0 as
select  
100 as code,
'Morning Walk' as descript,
SDO_GEOMETRY(
                 2002,                                         -- 2002 - two dimensional line
                 4326,                                         -- our usual coordinate system for lon/lat
                 NULL, -- ONLY for points
                 SDO_ELEM_INFO_ARRAY(1, 2, 1),    --line
                 SDO_ORDINATE_ARRAY(    -- lon/lat values
                    34.809512,    32.099221, 
                    34.810434,    32.099269,
                    34.810434,    32.099999
                 )
            ) as geometry
from dual;

Check it works fine:


And create the spatial index:

Step 3: Before creating a spatial index, we must insert a row of spatial metadata.

INSERT INTO USER_SDO_GEOM_METADATA VALUES (
  'Line0',  -- table name
  'GEOMETRY',    -- geometry column name
  SDO_DIM_ARRAY(
    SDO_DIM_ELEMENT('X', -180, 180, 0.005),
    SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),
   4326           -- indicates longitude/latitude coordinates (
coordinate system)
 );

Step 4: Create a spatial Index for performance

CREATE INDEX Line0_SIDX ON
      Line0 (
          GEOMETRY
      )
          INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;


Line GeoJSON based example

GeoJSON (see at https://geojson.org/) is a standard in the industry. 

GeoJson structure from LiveLabs workshop

Naturally, GeoJSON is still a JSON file. Oracle has now many JSON oriented options. There is an Autonomous JSON Database in Oracle cloud or simple drag and drop UI in "regular" Autonomous Database for JSON files loading. Yet, we have basic JSON support even in earlier versions such as 12c.

 In the Spatial Data LiveLab here (Lab 3) we have an example of loading data from files ans setting region tables in Autonomous Database. I will do a simple manual version of that process, that is less trivial that what is done in the Autonomous example of the Tornado Path file.

So our JSON file has the following structure:


 We want to load a JSON file into the database.

The JSON file is located in some storage that is mapped in the DB to a Directory. Access to directory required by spatial user to load JSON File. If needed system user / Admin should run something like:

GRANT read on DIRECTORY TEMP_DIR to user_name;
 

I will create a table named Line_GEOJSON that contains a single clob column named CLOB_JSON:

 Load the file tornado_paths.geojson

declare
    b bfile := bfilename('TEST_DIR','tornado_paths.geojson');
    c clob;
  l_dest_offset   integer := 1;
  l_src_offset    integer := 1;
  l_bfile_csid    number  := 0;
  l_lang_context  integer := 0;
  l_warning       integer := 0;
  begin
    dbms_lob.open(b);
    dbms_lob.createtemporary(c,true);
    dbms_lob.loadclobfromfile (
    dest_lob      => c,
    src_bfile     => b,
    amount        => dbms_lob.getlength(b),
    dest_offset   => l_dest_offset,
    src_offset    => l_src_offset,
    bfile_csid    => l_bfile_csid ,
    lang_context  => l_lang_context,
    warning       => l_warning);
    insert into Line_GEOJSON values (c);
    commit;
    dbms_lob.fileclose(b);
    dbms_lob.freetemporary(c);
  end;
  /

Next I can create table Line1 based on the JSON file data structure:

Create table Line1 as
select j.*
    from    Line_geojson,
            json_table(CLOB_JSON, '$.features[*]'  
                columns (
                    type    varchar2(128) path '$.type',                   
                    geometry SDO_GEOMETRY PATH '$.geometry',
                    YR NUMBER path '$.properties.YR',
                    LOSS NUMBER path '$.properties.LOSS',
                    L_Key NUMBER path '$.properties.KEY'
            )) j;

And check the result:

Finally create the spatial index:

Step 3: Before creating a spatial index, we must insert a row of spatial metadata.

INSERT INTO USER_SDO_GEOM_METADATA VALUES (
  'Line1',  -- table name
  'GEOMETRY',    -- geometry column name
  SDO_DIM_ARRAY(
    SDO_DIM_ELEMENT('X', -180, 180, 0.005),
    SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),
   4326           -- indicates longitude/latitude coordinates (
coordinate system)
 );

Step 4: Create a spatial Index for performance

CREATE INDEX Line1_SIDX ON
      Line1 (
          GEOMETRY
      )
          INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;

 

Lets check the length of each tornado path using the SDO_GEOM.SDO_LENGTH function:

 select
    L_Key,
    YR,
    SDO_GEOM.SDO_LENGTH(geometry,0.005, 'unit=KM') Length
from Line1;

 




Monday, August 7, 2023

Oracle Spatial - loading geographic data - Area

 Following the previous post - Oracle Spatial - SQL based Geographic data - Intro, few examples of loading data polygons/areas of geographical data.

I will cover here:

  • Manual creation
  • Using a string 
  • Loading GeoJSON

Polygon/Area Simple Example - Manual creation 

Step 1: Lets assume I have a table area with 2 columns (ID, Area_Name)  and 2 lines 

Step 2:
I will add Geometry column to Area1 table:

alter table Area1 add (geometry SDO_GEOMETRY);

and update a data row:

update AREA1
set geometry = SDO_GEOMETRY(
                 2003,                                         -- 2003 - two dimensional polygon
                 4326,                                         -- our usual
coordinate system for lon/lat
                 NULL, -- ONLY for points
                 SDO_ELEM_INFO_ARRAY(1, 1003, 1),    -- Signifies simple exterior polygon,
                 SDO_ORDINATE_ARRAY(    -- lon/lat values
                    34.809512,    32.099221, 
-- First point and last point must be the same                                                   34.809737,    32.098608,
                    34.809225,    32.098161,
                    34.809683,    32.09664,
                    34.81074,    32.097122,
                    34.813604,    32.097365,
                    34.813782,    32.098037,
                    34.81293,    32.098534,
                    34.811657,    32.098696,
                    34.810434,    32.099269,
                    34.809512,    32.099221  -- First point and last point must be the same
                 )
           )
where id=1                    -- Updating only the Pond row
;

In case you are interested, the maximum SDO_ORDINATE_ARRAY size is 1,048,576 numbers.

I will verify with select statement: 

 


Next I will update the other row and run step 3 and step 4 for indexing.

Step 3: Before creating a spatial index, we must insert a row of spatial metadata. 

INSERT INTO USER_SDO_GEOM_METADATA VALUES (
  'AREA1',  -- table name
  'GEOMETRY',    -- geometry column name
  SDO_DIM_ARRAY(
    SDO_DIM_ELEMENT('X', -180, 180, 0.005),
    SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),
   4326           -- indicates longitude/latitude coordinates (
coordinate system)
 );

Step 4: Create a spatial Index for performance

CREATE INDEX AREA1_SIDX ON
      AREA1 (
          GEOMETRY
      )
          INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;

 

 

Polygon/Area Simple Example - Using points (Long/Lat) String

We could do it better if we had a function that loads a string (CLOB better due to size constraints) of Longitude/Latitude sets and return SDO_GEOMETRY Polygon (2003,4326 like in the previous case)? Albert Godfrind in this Stack Overflow thread created great solution:

He created a function that does exactly that.

create or replace function string_to_geom (str clob)
return sdo_geometry
is
  s clob := str||',';
  i number;
  j number;
  t sdo_ordinate_array := sdo_ordinate_array();
begin
  i := 1;
  loop
    j := instr(s, ',', i);
    exit when j = 0;
    t.extend();
    t(t.count) := to_number(substr(s,i,j-i));
    i := j+1;
  end loop;
  return sdo_geometry (2003, 4326, null, sdo_elem_info_array (1,1003,1), t);
end;
/
show errors

 Now if I want to update the second row in my Area1 Table I can do the following:

update Area1
set GEOMETRY =
string_to_geom('34.813169,32.105909,34.813792,32.105355,34.808766,32.102542,34.808417,32.103285,34.813169,32.105909')
where id=2;


Now both of the rows are fine.

Steps 3 & 4 were done in the previous section.

 

Polygon/Area GeoJSON based example

 GeoJSON (see at https://geojson.org/) is a standard in the industry. 

GeoJson structure from LiveLabs workshop

Naturally, GeoJSON is still a JSON file. Oracle has now many JSON oriented options. There is an Autonomous JSON Database in Oracle cloud or simple drag and drop UI in "regular" Autonomous Database for JSON files loading. Yet, we have basic JSON support even in earlier versions such as 12c.

 In the Spatial Data LiveLab here (Lab 3) we have an example of loading data from files ans setting region tables in Autonomous Database. I will do a simple manual version of that process, that is less trivial that what is done in the Autonomous example. Here is an example how to do it by Jeff Kemp.

Unlike Jeff, I will set the geometry column as SDO_GEOMETRY and not JSON.

So our JSON file has the following structure:


 

We want to load a JSON file into the database. First you can check the data using http://geojson.io site. Just drop the file on the map.



The JSON file is located in some storage that is mapped in the DB to a Directory. Access to directory required by spatial user to load JSON File. If needed system user / Admin should run something like:

GRANT read on DIRECTORY TEMP_DIR to user_name;
 

I will create a table named Region_GEOJSON that contains a single clob column named features:

create table Region_GEOJSON ( features clob, constraint RG_chk check ( features is json ));

Then load the file regions.geojson

declare
    b bfile := bfilename('TEST_DIR','regions.geojson');
    c clob;
  l_dest_offset   integer := 1;
  l_src_offset    integer := 1;
  l_bfile_csid    number  := 0;
  l_lang_context  integer := 0;
  l_warning       integer := 0;
  begin
    dbms_lob.open(b);
    dbms_lob.createtemporary(c,true);
    dbms_lob.loadclobfromfile (
    dest_lob      => c,
    src_bfile     => b,
    amount        => dbms_lob.getlength(b),
    dest_offset   => l_dest_offset,
    src_offset    => l_src_offset,
    bfile_csid    => l_bfile_csid ,
    lang_context  => l_lang_context,
    warning       => l_warning);
    insert into Region_GEOJSON values (c);
    commit;
    dbms_lob.fileclose(b);
    dbms_lob.freetemporary(c);
  end;
  /

 and see the data:

Next I can create table Regions that 

create table Regions that parses the JSON file:
select j.*
    from    Region_geojson,
            json_table(features, '$.features[*]'  -- features is the name of my CLOB column in the prev. table                                                                      -- while '$.features[*]' is the standard syntax
                columns (
                    type    varchar2(128) path '$.type',                   
                   
geometry SDO_GEOMETRY PATH '$.geometry',
                    region  varchar2(256) path '$.properties.REGION'
            )) j;

 this is the result:

 


Now we have our data and spatial column in a table the next steps are similar to what we did before... Update metadata and create an index:

  1. Have you data in a table.
  2. Set Spatial column.
  3. Update the USER_SDO_GEOM_METADATA table about the new spatial data source.
  4. Set Spatial Index.

 INSERT INTO USER_SDO_GEOM_METADATA VALUES (
  'REGIONS',  -- table name
  'GEOMETRY',    -- geometry column name
  SDO_DIM_ARRAY(
    SDO_DIM_ELEMENT('X', -180, 180, 0.005),
    SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),
   4326           -- indicates longitude/latitude coordinates (coordinate system)
 );


CREATE INDEX REGIONS_SIDX ON
      REGIONS (
          GEOMETRY
      )
          INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;

Now I can check the size of each region:

Select
    REGION,
     SDO_GEOM.SDO_AREA(geometry, 0.005, 'unit=SQ_KM') Reg_area
From Regions;


The full list of units of measures can be obtained using the following select:

 select distinct short_name, unit_of_meas_name
from sdo_units_of_measure
where unit_of_meas_type = 'area';

 

And last tip... Having problems with your data? SDO_UTIL.RECTIFY_GEOMETRY can fix various problems in your geometries.

See here additional Recommendations for Loading and Validating Spatial Data.

Oracle Spatial - loading geographical data - Points

Following the previous post - Oracle Spatial - SQL based Geographic data - Intro, few examples of loading data points of geographical data.

I will show 2 examples:

a simple case and using a function instead of creating specific SDO column.

Data points example

Step 1: Lets assume we have a Warehouse table with warehouse name, Latitude and Longitude.

Step 2: Lets create a column  named Geometry with the Long & Lat column set as spatial data.  I will use the SDO_GEOMETRY function. 

We can alter the table and add a column named Geometry:

 ALTER TABLE WAREHOUSES ADD (
    GEOMETRY SDO_GEOMETRY
    );

Next update it using SDO_GEOMETRY function while passing the 2 columns as parameters:

UPDATE WAREHOUSES
SET GEOMETRY = SDO_GEOMETRY(
                 2001,                                                             -- 2001 - point
                 4326,                                                             -- 4326 - Longitude & Latitude WSG 84 system
                 SDO_POINT_TYPE(LON, LAT, NULL),      -- LON & LAT columns, the null is for  3-DIM   
                 NULL, NULL);                                              -- used for lines / polygons.

Actually, we don't want wrong data to be inserted so the following is better:

UPDATE WAREHOUSES
SET GEOMETRY = SDO_GEOMETRY(
                 2001,
                 4326,
                 SDO_POINT_TYPE(LON, LAT, NULL),
                 NULL, NULL)
WHERE LON IS NOT NULL
AND LON BETWEEN -180 AND 180
AND LAT IS NOT NULL
AND LAT BETWEEN -90 AND 90 ;

Step 3:  -- Before creating a spatial index, we must insert a row of spatial metadata. Every user has an updatable view called USER_SDO_GEOM_METADATA for their spatial metadata.  This is a user view on a centralized table storing spatial metadata for the entire database instance. 

 INSERT INTO USER_SDO_GEOM_METADATA VALUES (
  'WAREHOUSES',  -- table name
  'GEOMETRY',    -- geometry column name
  SDO_DIM_ARRAY(
    SDO_DIM_ELEMENT('X', -180, 180, 0.005),  -- 'X',[min x],[max x],[tolerance]
    SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),    -- 'Y',[min y],[max y],[tolerance]
   4326           -- indicates longitude/latitude coordinates (
coordinate system)
 );

The tolerance value (0.005 here) represents the level of precision. For geodetic data such as the one we have here, the value is in meters. So 0.005m = 0.5 cm. Meaning if distance between points less than 0.5cm they are considered the same. For non geodetic data it represents the unit that is used.

Step 4: Create a spatial Index for performance

CREATE INDEX WAREHOUSES_SIDX ON
      WAREHOUSES (
          GEOMETRY
      )
          INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;

If a spatial index creation statement fails (for example because a previous step was not done correctly), then before retrying you must drop the index since some index artifacts may have been created.
In our case before retrying you should run "DROP INDEX WAREHOUSES_SIDX;".

Please notice that as a result you might discover a table such as  MDRT_1BB0E$ created in the schema. Such tables are managed entirely by Spatial to support spatial indexes and should never be manually dropped. Database users should ignore them.


Data point Example with Function instead of Physical Geometry column

In Oracle DB we can decide to leave the longitude/latitude columns in the original state and use a function instead of physical Geometry column. Please note that such a solution might have performance implications.

Lets repeat the previous Data Points example with a function instead of physical Geometry column.

I will create a table warehouse_f

create table warehouses_F as
select Warehouse_name, Lat, Lon
from WAREHOUSES

and a Generic Function GET_GEOMETRY.

It accepts coordinates and returns a geometry (i.e., a SDO_GEOMETRY value).
The function includes criteria so that a result is only returned for valid input coordinates.

CREATE OR REPLACE FUNCTION GET_GEOMETRY (
      IN_LONGITUDE NUMBER,
      IN_LATITUDE  NUMBER
  ) RETURN SDO_GEOMETRY
      DETERMINISTIC PARALLEL_ENABLE
  IS
  BEGIN
   IF (IN_LONGITUDE IS NOT NULL
      AND IN_LONGITUDE BETWEEN -180 AND 180
      AND IN_LATITUDE IS NOT NULL
      AND IN_LATITUDE BETWEEN -90 AND 90)
   THEN
    RETURN
      SDO_GEOMETRY(
        2001,
        4326,
        SDO_POINT_TYPE(IN_LONGITUDE, IN_LATITUDE, NULL),
        NULL, NULL);
    ELSE RETURN NULL;
    END IF;
  END;
  /

If you want, you can test the function GET_GEOMETRY (SDO_UTIL.TO_GEOJSON function makes the spatial data readable):


 

If we return to the 4 steps of  setting Spatial data. We got 1& 2 covered.

  1. Have you data in a table.
  2. Set Spatial column.
  3. Update the USER_SDO_GEOM_METADATA table about the new spatial data source.
  4. Set Spatial Index.

Step 3: Before creating a spatial index, we must insert a row of spatial metadata. The schema/user I'm working here is name SPATIAL

 INSERT INTO USER_SDO_GEOM_METADATA VALUES (
 'WAREHOUSES_F',  -- table name
 'SPATIAL.GET_GEOMETRY(LON,LAT)', --
geometry function. Schema name (spatial) is critical
 SDO_DIM_ARRAY(
  SDO_DIM_ELEMENT('X', -180, 180, 0.005),
  SDO_DIM_ELEMENT('Y', -90, 90, 0.005)),
 4326  -- indicates longitude/latitude coordinates
);

Step 4: Create a spatial Index for performance. In the case of a function-based spatial index, the "column" being indexed is actually the call to the GET_GEOMETRY function.


CREATE INDEX WAREHOUSES_F_SIDX ON
  WAREHOUSES_F ( -- the table name
      GET_GEOMETRY(LON,LAT)  -- the function call instead of the column name
  )
      INDEXTYPE IS MDSYS.SPATIAL_INDEX_V2;



Oracle Spatial - SQL based Geographic data - Intro

 Oracle Spatial is a free component of various versions of Oracle database, including on-premises or cloud versions and of course the Autonomous Oracle Database. 

The Spatial option helps developers and analysts get started easily with location intelligence analytics and mapping services. It enables Geographic Information System (GIS) professionals to successfully deploy advanced geospatial applications. 

In the later post I will talk about Spatial Studio and an easy, no code, UI (User Interface) to work with spatial data. I will start with few core in-database posts first.

Few links:

Here is the spatial page from Oracle.

Here are the links to the documentation of release 19 and release 21 of the Developer Guide.

Oracle LiveLabs training of "Work with Spatial Data in Oracle Autonomous Database".

Spatial Performance Doc is here.

Geocoder Guide (Converting Address to geographic coordinates - Long/Lat) is here.

 

With Oracle Spatial we can run SQL statement such as:

Show me stores within 30 kilometers of a specific warehouse

 

Find Regions that have any interaction with Tornado Paths
 


Tedious Intro with Links

Oracle Spatial supports:


 

 various types of spatial data: 

  • Vector data ⎻ Points, Lines, Polygons
  • Raster data ⎻ Digital Imagery and Gridded Data
  • GPS Tracking data ⎻ For Coinciding track analysis / GeoFence analysis
  • LIDAR Data ⎻ Point Cloud / LIDAR data
  • Network Model ⎻ Drive Time / Connectivity Analysis

Various types of Geometry (from point, lines and polygons to multi-polygons, solid/cuboid and multi-solid).  Here is a list of SGO_GTYPEs of 3D spatial objects.

Various types of Coordination Systems such as Cartesian, Geodetic, Projected, Local...

Various types of Operators. For example few operators for, the most common, vector data: 

Topological Relations


Operators


SDO_GEOM package contains subprograms for working with geometry object to check Relationship, Validations and single/two-object operations...  

  • Relationship (True/False) between two objects: RELATE, WITHIN_DISTANCE

  • Validation: VALIDATE_GEOMETRY_WITH_CONTEXT, VALIDATE_LAYER_WITH_CONTEXT, SDO_SELF_UNION

  • Single-object operations: SDO_ALPHA_SHAPE, SDO_ARC_DENSIFY, SDO_AREA, SDO_BUFFER, SDO_CENTROID, SDO_CONVEXHULL, SDO_CONCAVEHULL, SDO_CONCAVEHULL_BOUNDARY, SDO_DIAMETER, SDO_DIAMETER_LINE, SDO_LENGTH, SDO_MBC, SDO_MBC_CENTER, SDO_MBC_RADIUS, SDO_MAX_MBR_ORDINATE, SDO_MIN_MBR_ORDINATE, SDO_MBR, SDO_POINTONSURFACE, SDO_TRIANGULATE, SDO_VOLUME, SDO_WIDTH, SDO_WIDTH_LINE

  • Two-object operations: SDO_CLOSEST_POINTS, SDO_DISTANCE, SDO_DIFFERENCE, SDO_INTERSECTION, SDO_MAXDISTANCE, SDO_MAXDISTANCE_LINE, SDO_UNION, SDO_XOR

 


 Working with SQL - Creating Table and Index

 Basically they are 4 steps setting Spatial data.

  1. Have you data in a table.
  2. Set Spatial column.
  3. Update the USER_SDO_GEOM_METADATA table about the new spatial data source.
  4. Set Spatial Index.

Steps 1&2 can be combined into one. 

We will use SDO_Geometry function to set spatial data in a column.

SDO_GEOMETRY( [geometry type] -- ID for points/lines/polygons , [coordinate system] -- ID of coordinate system , [point coordinate] -- used for points only , [line/polygon info] -- used for lines/polygons only , [line/polygon coordinates] -- used for lines/polygons only )

For example:

geometry type = 2001. The 2 indicates two-dimensional, and the 1 indicates a single point. Two-dimensional examples we will use:

IDType
2001Point
2002Line
2003Polygon

coordinate system =4326. Represents spatial data using longitude and latitude coordinates on the Earth's surface. 

there are several thousand options of Coordination Systems we can use. You can list all of them using the following SQL:

 SELECT srid, COORD_REF_SYS_NAME from sdo_coord_ref_sys order by srid;

 

Next I will post examples of all 3 types (Points, Line and Polygon) with some variations listed bellow:

  • Data points 
    • Example (2 columns of Longitude/Latitude)
    • Example with Function instead of Physical Geometry column
  • Polygon/Area 
    • Simple Example - Manual creation 
    • Simple Example - Using a list of points (Long/Lat) String
    • JSON based example
  • Line 
    • Simple Example - Manual creation 
    • JSON based example